It seems like the way that ServerTransport is batching the objects and sending to POST /objects/:streamId is not able to handle a large amount of objects, because it yields enough POST requests to that endpoint, that it restarts returning 500 errors.
The solution would be for the code on the sandbox mentioned above to work.
It works (the objects are sent)
It is fast. I am under the impression that due to the 500 errors the retry mechanism is just making the process hang before attempting again.
Technically, I would suggest enhancing the ServerTransport to compress the batch using application/zip instead of application/json, so that it can send more objects per request, meaning less total requests.
linear[bot]
commented
3 months ago
It seems like the way that
ServerTransportis batching the objects and sending to POST/objects/:streamIdis not able to handle a large amount of objects, because it yields enough POST requests to that endpoint, that it restarts returning 500 errors.@iainsproat mention it can be a firewall or other mechanism.
Problem
I have described the problem here.
Also, I have produced a codesandbox to reproduce the issue here.
Solution
The solution would be for the code on the sandbox mentioned above to work.
Technically, I would suggest enhancing the
ServerTransportto compress the batch usingapplication/zipinstead ofapplication/json, so that it can send more objects per request, meaning less total requests.Additional context
Please have a look at the following discussion: https://speckle.community/t/is-there-a-rate-limit-on-post-objects-projectid/12310