Timeout in Client library for 3.0 SX library means we can't submit a very large file request

[gordonwatts]

SX takes close to a minute to submit a large request, however the current timeout for submission is 5 seconds.

• Increase timeout in the event
• Understand what in SX is causing the delay.

[gordonwatts]

The timeout change is being worked on here: https://github.com/ssl-hep/ServiceX_frontend/pull/369

[gordonwatts]

Looks like this code in SX is the culprit - or downstream.

There is what I think happens:

1. Request comes in for the 50 TB data sample.
2. The DID finder goes to see if we know about the sample.
3. In this case we do, so it extracts it from the database. This is 65K files.
4. This (and putting them into another database downstream?) takes some 45 or 50 seconds.
5. The client http request code times out (5 second timeout for client requests).

[gordonwatts]

The timeout fix was done and merged. Closing this unless the problem re-appears!