benbierens
commented
3 months ago When you POST to /storage/request/{cid}, the following work is being done by the application:
- Erasure coding: expands the dataset (if you have 5 nodes + 5 tolerance, the dataset will be 2x as large after EC)
- Slot building: Organize the expanded dataset into a number of slots.
- For each slot, we build a new merkle tree with poseidon2 hashing (which is actually built on top of tiny poseidon2-trees of each block)
- With all the slot roots, we build a new dataset root with poseidon2
The hashing and erasure-coding a CPU-intense operations. None of this is being performed on a separate thread. If you have a large dataset, this process might take a while and the app will be completely unresponsive during that time. This is a real problem, not just for users but also: during this time the app doesn't respond to network traffic. Other nodes may consider your node lost, and drop you from their routing tables.
Needless to say, this is something client-team has to fix. An obvious approach would be to move the erasure-coding/slot-building to another thread. Ideally, the API would have a "work object" that represents the inputs, outputs, and status of this entire process, so that the user could have some feedback (and the POST itself wouldn't need to block till it's finished).
gmega
Describe the bug I uploaded an MP4 file of 583.7 MB size. When I am trying to create a storage request for this file using the REST API, the codex client hangs and I am not able to make any requests anymore.
To Reproduce Steps to reproduce the behavior:
Expected behavior The storage request is created and I receive the request ID. I am expecting not to be blocked while making the request.
Environment:
Additional context When the storage request expires, the client is not hanging anymore.
When I kill (Ctrl + C) the client during a hanging phase, the client crashes with these logs.