search

credo-science / credo-webapp

Credo web application
MIT License
2 stars 6 forks source link

data export #31

Closed authule closed 6 years ago

authule commented 6 years ago

Hello,

1) Could you please suggest why wget command does not work from a script with the links generated for data export while working correctly from a shell (not stable, though)?

Resolving storage.cloud.cyfronet.pl (storage.cloud.cyfronet.pl)... 149.156.9.155, 149.156.9.154 Connecting to storage.cloud.cyfronet.pl (storage.cloud.cyfronet.pl)|149.156.9.155|:443... connected. HTTP request sent, awaiting response... 404 Not Found 2018-06-14 16:00:07 ERROR 404: Not Found.

2) Why when downloading the "ping" information the ping category itself if filled mostly with zeros? Does it mean the ping value isn't registered correctly?

3) The on-time field contains only zeros. Is its calculation not implemented yet?

4) Could anything be suggested to avoid requests' throttling (at lease for a short period)?

Thanks in advance, Alex

koziomek commented 6 years ago
  1. It seems like you were trying to download the exported data before the export process was finished. After requesting data export we start preparing it in the background. After the process is finished it will be uploaded to the URL that was sent in the /api/v2/data_export response.

Try waiting a few minutes before trying to download data or implement retry logic on 404.

2, 3. It seems like the mobile detector application is sending these values, we are only saving them.

  1. I'll raise them a bit. However, even 3 requests are enough to get the whole data set. Why exactly do you need more?
authule commented 6 years ago

Thanks for your response, Krzysztof,

4) the problem was related to multiple wget attempts, described in 1. However, your explanation helped with both issues, so there is no urgency of changing throttling condition any more.

Thanks again!

koziomek commented 6 years ago

It seems like people responsible for the app fixed the bug with not sending on_time. https://github.com/credo-science/credo-detector-android/commit/88d24caa7a14f6f09e1c5796722a06f2c0fc7e19

If you have any more questions please ask.