search

Quansight / lsst_dashboard

LSST Dashboard https://quansight.github.io/lsst_dashboard/
BSD 3-Clause "New" or "Revised" License
8 stars 3 forks source link

Reconsider the CI sample dataset #126

Closed kcpevey closed 4 years ago

kcpevey commented 4 years ago

The CI runs the download script to allow us to use the data for testing. However, it has been failing 1 out of 6 times recently. It looks like the download rates are fairly slow (200k). The curl command failed after taking 30 minutes to download 300MB of data. We should look into why the download speeds are so slow or using a smaller test dataset.

kcpevey commented 4 years ago

https://github.com/Quansight/lsst_dashboard/pull/124/checks?check_run_id=488170658

kcpevey commented 4 years ago

I can confirm that download speeds of these files on my local machine are normal so it seems to be limited to github actions env

kcpevey commented 4 years ago 
Troubleshooting slow downloads and intermittent slow connections
GitHub does not throttle bandwidth per user.

If you're experiencing slow connections at certain times of day but not others, 
the slow speeds are  most likely due to network congestion. Because GitHub   
cannot resolve network congestion, you should escalate the problem to   
your internet service provider.

https://help.github.com/en/github/getting-started-with-github/troubleshooting-connectivity-problems

timothydmorton commented 4 years ago

That's not particularly helpful.. :-( I'd hang tight on worrying about this for now until we get a test dataset with the newly partitioned data. If all is good with what @dharhas was doing, then these datasets will be significantly smaller. Also, if we really need to make a tiny test dataset for CI, we can do that, too.

kcpevey commented 4 years ago

Closing since using KTK is changing things. Also #144 discusses the KTK CI dataset