xtracthub / xtract-service

Globus Labs Xtract: Extract metadata from distributed data sets.
6 stars 1 forks source link

Built-in funcX rate limiting below 3000 requests per 5 mintues. #58

Open tskluzac opened 3 years ago

tskluzac commented 3 years ago

Right now I'm running the scaling tests on Xtract (for all of MDF) and it seems like we fall pretty well below 3000 (~1200) requests per 5 minutes by manually changing the number of threads. This is not conducive to unique network conditions.

I propose that we make this internally configurable to m and n% where m% of requests are for sending and n% are for polling.