search

Deadwood-ai / deadwood-api

Main FastAPI application for the deadwood backend
GNU General Public License v3.0
0 stars 0 forks source link

Implement download functionality #19

Closed JesJehle closed 3 months ago

JesJehle commented 3 months ago
cmosig commented 3 months ago

Your call on the rate-limit, but why not full-send until it fails? I.e. not optimize issues that we dont have yet.

mmaelicke commented 3 months ago

We kind of already have that issue. The problem is the shared infrastructure. If someone uses a script to download (which are thousands of concurrent downloads and we have no protection against that, the server does not respond anymore and the website cannot be used anymore. Thus we need to assign resources in a meaningful way. For the upload we implemented a very simple queue, for the download we need a rate limit. That way, one core will always be free to serve all other requests.

@JesJehle is this one the issue with higest prio?

cmosig commented 3 months ago

I see, so there is no bandwidth rate-limit?

Can one then iterate through all files without hitting the rate-limit and request file X, wait for download of file X, request file Y, ...?

mmaelicke commented 3 months ago

That would be technically possible. The question is then, if we want to encourage it.

That being said, for internal use (training, label prediction) or based on email (you, teja) we can always bypass limits and/or prioritize ourselves.

cmosig commented 3 months ago

Hmm, generally, I think it would be best to maximise what an individual user can download as long as the infrastructure can handle it, until it becomes an issue.

I have the feeling that ease-of-use and fast access increases usage of products and generally rate-limits can increase frustration.