Rebalancing doesn't work and causes the server to crash.

[argahsuknesib]

Hello, I am loading up a .nt file with a size of 2.1GB and then sorting works fine but I can't load it up on the Solid Pod. What could be wrong as the other side is closed? Is it a CSS server issue or the library issue?

[svrstich]

Hi Kush, indeed. I experienced this before. Problem looks to be too many requests on too short a timeframe. Seems to be linked to the library used to execute the GET/POST (undici). We had a discussion about this earlier on Mattermost as well (https://mattermost.ilabt.imec.be/idlab/pl/5sthgf6kybgpuypu6pugrzrjkh)

[argahsuknesib]

Hello stijn. Yes indeed, I have faced similar issues due to the large amount of HTTP requests being done to the Solid Server while requesting the sensor events from the files with HTTP GET. I rate limited the requests for the server to be able to handle. What do you think of rate limiting the replayer so that it is indeed slow(er) but does not crash?

[svrstich]

For sure ... but I thought I had already provided for some mechanisms to do this, but apparently they have remained in the ideation/early implementation phase, rather than being properly implemented. :-s

[argahsuknesib]

I've implemented rateLimitedGet request here for my implementation of the aggregator at https://github.com/SolidLabResearch/solid-stream-aggregator/blob/master/src/utils/ldes-in-ldp/RateLimitCommunication.ts

I can make it a package with other requests if it's usable for you.

[svrstich]

Sounds like a good idea 😉

From: Kush @.> Sent: Monday, 6 November 2023 18:16 To: SolidLabResearch/LDES-in-SOLID-Semantic-Observations-Replay @.> Cc: Stijn Verstichel (UGent-imec) @.>; Comment @.> Subject: Re: [SolidLabResearch/LDES-in-SOLID-Semantic-Observations-Replay] Rebalancing doesn't work and causes the server to crash. (Issue #24)

I've implemented rateLimitedGet request here for my implementation of the aggregator at https://github.com/SolidLabResearch/solid-stream-aggregator/blob/master/src/utils/ldes-in-ldp/RateLimitCommunication.ts

I can make it a package with other requests if it's usable for you.

— Reply to this email directly, view it on GitHubhttps://github.com/SolidLabResearch/LDES-in-SOLID-Semantic-Observations-Replay/issues/24#issuecomment-1795529734, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AETEJFTTNCSUFBM2BIQBCKDYDELMJAVCNFSM6AAAAAA6RE3ROOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOJVGUZDSNZTGQ. You are receiving this because you commented.Message ID: @.**@.>>

[argahsuknesib]

Hello stijn, I found this library which already implemented the rate limiting communication. I think you might have to rewrite the methods from the existing LDES client to make them rate limited.

https://github.com/aishek/axios-rate-limit