search

cshneider / soho-ml-data-ready

5 stars 3 forks source link

VSO query issues when trying out #2

Open sandyfreelance opened 2 years ago

sandyfreelance commented 2 years ago

Hi-- trying to run it, hit two issues out of the box. I am trying a sample query to test this out. The primary error is "has no attribute 'get_response'" from the query, where it looks like it's fetching VSO data but unable to parse it. There was also a SunPy deprecation bug that I worked around (which may or may not be part of the problem). Help appreciated.

Here's the invocation: python Mission_Data_Gen.py --mission SOHO --products EIT195 --home_dir ~/temp --fits_headers y --date_start 2013-01-01 --date_finish 2013-01-03 --image_size_output 512 --flag subsample --time_window 1 --email sandy.antunes@jhuapl.edu --lev1_LASCO y

First problem, 

date_time_start: 2013-01-01 00:00:00
date_time_end: 2013-01-03 23:59:00
time increment 60 days
image_size_output: 512
flag: subsample
home_dir: /Users/antunak1/temp
url_prefix: https://seal.nascom.nasa.gov/
look_ahead: 6
diff_start_finish_total_sec: 259140.0
total_sec: 5184000.0
num_loops: 2.0
  0%|                                                     | 0/1 [00:00<?, ?it/s]EIT195
                                                                               t_value: 0.0                                               | 0/2 [00:00<?, ?it/s]
prev_time: []
wavelen: 195
  0%|                                                     | 0/2 [00:00<?, ?it/s]
  0%|                                                     | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "Mission_Data_Gen.py", line 165, in <module>
    main(
  File "Mission_Data_Gen.py", line 88, in main
    product_results, client = product_search(base, time_range_modified, client, mission, time_window) 
  File "/Users/antunak1/t/soho-ml-data-ready/Mission_utility/__init__.py", line 456, in product_search
    product_results = Fido.search(a.vso.Time(time_range.start,time_range.end), a.vso.Source(f'{mission}'), a.vso.Instrument('EIT'), a.vso.Provider('SDAC'), a.vso.Wavelength(wavelen * a.vso.u.Angstrom, wavelen * a.vso.u.Angstrom))
AttributeError: module 'sunpy.net.vso.attrs' has no attribute 'Time'

A few fixes (ala https://github.com/sunpy/sunpy/issues/3814) later, 

       #product_results = Fido.search(a.vso.Time(time_range.start,time_range.e\
nd), a.vso.Source(f'{mission}'), a.vso.Instrument('EIT'), a.vso.Provider('SDAC'\
), a.vso.Wavelength(wavelen * a.vso.u.Angstrom, wavelen * a.vso.u.Angstrom))    
    product_results = Fido.search(a.Time(time_range.start,time_range.end), \
a.Source(f'{mission}'), a.Instrument('EIT'), a.Provider('SDAC'), a.Wavelength(w\
avelen * u.Angstrom, wavelen * u.Angstrom))

Now I get the error:

Traceback (most recent call last):
  File "Mission_Data_Gen.py", line 165, in <module>
    main(
  File "Mission_Data_Gen.py", line 104, in main
    ind, fits_headers = index_of_sizes(base,product_results, fits_headers, lev1_LASCO, mission) #putting fits_headers here to insert it into __init__.py
  File "/Users/antunak1/gits/soho-ml-data-ready/Mission_utility/__init__.py", line 531, in index_of_sizes
    size_list = [elem['size'] for elem in product_results.get_response(0)[:]]
AttributeError: 'UnifiedResponse' object has no attribute 'get_response'

I added in 2 debugging lines:

def index_of_sizes(base,product_results,fits_headers,lev1_LASCO,mission):
    print("DEBUG",product_results,"end DEBUG")
    print("DEBUGDIR",dir(product_results),"end DEBUGDIR")

and the result is:

python Mission_Data_Gen.py --mission SOHO --products EIT195 --home_dir ~/temp --fits_headers y --date_start 2013-01-01 --date_finish 2013-01-03 --image_size_output 512 --flag subsample --time_window 1 --email sandy.antunes@jhuapl.edu --lev1_LASCO y

date_time_start: 2013-01-01 00:00:00
date_time_end: 2013-01-03 23:59:00
time increment 60 days
image_size_output: 512
flag: subsample
home_dir: /Users/antunak1/temp
url_prefix: https://seal.nascom.nasa.gov/
look_ahead: 6
diff_start_finish_total_sec: 259140.0
total_sec: 5184000.0
num_loops: 2.0
  0%|                                                                                                                                                                                                         | 0/1 [00:00<?, ?it/s]EIT195
                                                                                                                                                                                                                                   t_value: 0.0                                                                                                                                                                                                   | 0/2 [00:00<?, ?it/s]
prev_time: []
wavelen: 195
DEBUG Results from 1 Provider:

6 Results from the VSOClient:
Source: http://vso.stanford.edu/cgi-bin/search
Total estimated size: 12.65 Mbyte

       Start Time               End Time        Source Instrument Wavelength [2] Provider  Physobs  Extent Type   Size 
                                                                     Angstrom                                    Mibyte
----------------------- ----------------------- ------ ---------- -------------- -------- --------- ----------- -------
2013-01-01 01:13:41.000 2013-01-01 01:13:53.000   SOHO        EIT 195.0 .. 195.0     SDAC intensity    FULLDISK 2.01074
2013-01-01 13:13:38.000 2013-01-01 13:13:50.000   SOHO        EIT 195.0 .. 195.0     SDAC intensity    FULLDISK 2.01074
2013-01-02 01:13:41.000 2013-01-02 01:13:53.000   SOHO        EIT 195.0 .. 195.0     SDAC intensity    FULLDISK 2.01074
2013-01-02 13:13:42.000 2013-01-02 13:13:54.000   SOHO        EIT 195.0 .. 195.0     SDAC intensity    FULLDISK 2.01074
2013-01-03 01:13:39.000 2013-01-03 01:13:51.000   SOHO        EIT 195.0 .. 195.0     SDAC intensity    FULLDISK 2.01074
2013-01-03 13:13:38.000 2013-01-03 13:13:50.000   SOHO        EIT 195.0 .. 195.0     SDAC intensity    FULLDISK 2.01074

 end DEBUG
DEBUGDIR ['__abstractmethods__', '__class__', '__contains__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__reversed__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '__weakref__', '_abc_impl', '_getitem_string', '_list', '_numfile', '_repr_html_', 'all_colnames', 'count', 'file_num', 'index', 'keys', 'path_format_keys', 'show'] end DEBUGDIR
  0%|                                                                                                                                                                                                         | 0/2 [00:02<?, ?it/s]
  0%|                                                                                                                                                                                                         | 0/1 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "Mission_Data_Gen.py", line 165, in <module>
    main(
  File "Mission_Data_Gen.py", line 104, in main
    ind, fits_headers = index_of_sizes(base,product_results, fits_headers, lev1_LASCO, mission) #putting fits_headers here to insert it into __init__.py
  File "/Users/antunak1/gits/soho-ml-data-ready/Mission_utility/__init__.py", line 531, in index_of_sizes
    size_list = [elem['size'] for elem in product_results.get_response(0)[:]]
AttributeError: 'UnifiedResponse' object has no attribute 'get_response'
Guemann-ui commented 1 year ago

@sandyfreelance I faced the same errors! I solved them except the last one (AttributeError: 'UnifiedResponse' object has no attribute 'get_response') even by using "sunpy.net.dataretriever.client Module" ...

Did you solve it please ?

Thanks,

sandyfreelance commented 1 year ago

Alas, no-- got in touch with the programmer (Carl) and was told "At the time that the code was written, the instructions on the Github worked fine. At that time, the code was thoroughly tested and debugged. Since the original code base isn't maintained, as there is no longer time nor funding for this effort, software incompatibilities will arise with time. In my opinion, it will be up to interested parties to make the appropriate adaptations to keep things moving forward."

Which is pretty common. With NASA saying all code must be released as open source, I hope they also consider how maintainence like this could use funding. Expecting code creators to maintain tools past a research grant is problematic.

Separately, my team is releasing a set of all SOHO EIT FITS files, calibrated to level 1, alongside the STEREO A & B EUVI files, suitable for machine learning. Our expected due date is August 2023 (that's when our grant end). It will be in the GSFC HelioCloud in AWS S3 storage, and NASA HDRL has said they'll mirror it for non-cloud users. I hope you're not in a hurry and that timeline works.