Closed troyraen closed 2 months ago
The new bucket is in a different region. How does this affect performance? Collecting results of time tests in this comment.
Test code, to be run from Fornax and ISP:
import sample_selection
import wise_functions
coords, labels = [], []
sample_selection.get_sdss_sample(coords, labels, num=100)
sample_table = sample_selection.clean_sample(coords, labels)
%time unwise_df = wise_functions.wise_get_lightcurves(sample_table, radius=1.0, bandlist=['W1', 'W2'])
Results (Wall time):
Bucket (region) | Fornax | ISP |
---|---|---|
irsa-mast-tike-spitzer-data | ||
(us-east-1) | 7min 37s | 16min 53s |
7min 20s | 18min 55s | |
6min 45s | 17min 58s | |
nasa-irsa-wise | ||
(us-west-2) | 9min 21s | 8min 36s |
8min 6s | 8min 14s | |
8min 6s | 7min 33s |
I believe Fornax is running in us-east-1 and ISP close to us-west-2. So, relative to hitting the close bucket, hitting the far bucket increases wall time by about a factor of about 1.2 on Fornax and 2.2 on ISP.
Thanks!
The unWISE dataset accessed by the light curve notebooks is now in a NASA ODR bucket. This PR updates the code to access that bucket.