nasa-fornax / fornax-demo-notebooks

Demo notebooks for the Fornax project
https://nasa-fornax.github.io/fornax-demo-notebooks/
BSD 3-Clause "New" or "Revised" License
9 stars 19 forks source link

Change unWISE bucket to nasa-irsa-wise #330

Closed troyraen closed 2 months ago

troyraen commented 2 months ago

The unWISE dataset accessed by the light curve notebooks is now in a NASA ODR bucket. This PR updates the code to access that bucket.

troyraen commented 2 months ago

The new bucket is in a different region. How does this affect performance? Collecting results of time tests in this comment.

Test code, to be run from Fornax and ISP:

import sample_selection
import wise_functions

coords, labels = [], []
sample_selection.get_sdss_sample(coords, labels, num=100)
sample_table = sample_selection.clean_sample(coords, labels)

%time unwise_df = wise_functions.wise_get_lightcurves(sample_table, radius=1.0, bandlist=['W1', 'W2'])

Results (Wall time):

Bucket (region) Fornax ISP
irsa-mast-tike-spitzer-data
(us-east-1) 7min 37s 16min 53s
7min 20s 18min 55s
6min 45s 17min 58s
nasa-irsa-wise
(us-west-2) 9min 21s 8min 36s
8min 6s 8min 14s
8min 6s 7min 33s

I believe Fornax is running in us-east-1 and ISP close to us-west-2. So, relative to hitting the close bucket, hitting the far bucket increases wall time by about a factor of about 1.2 on Fornax and 2.2 on ISP.

bsipocz commented 2 months ago

Thanks!