search

sunpy / sunpy

SunPy - Python for Solar Physics
http://www.sunpy.org
BSD 2-Clause "Simplified" License
908 stars 583 forks source link

XRT results from VSO fail to return because of date formatting #7244

Closed wtbarnes closed 11 months ago

wtbarnes commented 11 months ago

Describe the bug

XRT data is once again available from the VSO. However, querying this data with Fido fails when trying to return the response table. This seems to be due to the way the dates are formatted when they are returned. Looking at the XML body (see below), there are a few dates where there are spaces between the minute and second designation in the returned dates,

<time>
    <start>2020-11-09 14:03: 0.172</start>
    <end>2020-11-09 14:03: 0.895</end>
</time>

To Reproduce

import astropy.time
from sunpy.net import Fido, attrs as a
Fido.search(
    a.Time('2020-11-09 14:00:00', end='2020-11-09 14:30:00'),
    a.Instrument('XRT'),
)

throws the exception

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/core.py:597, in TimeBase._get_time_fmt(self, val, val2, format, scale, precision, in_subfmt, out_subfmt)
    596 try:
--> 597     return cls(val, val2, scale, precision, in_subfmt, out_subfmt)
    598 except UnitConversionError:

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/formats.py:159, in TimeFormat.__init__(self, val1, val2, scale, precision, in_subfmt, out_subfmt, from_jd)
    158 else:
--> 159     val1, val2 = self._check_val_type(val1, val2)
    160     self.set_jds(val1, val2)

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/formats.py:1055, in TimeDatetime._check_val_type(self, val1, val2)
   1054 if not all(isinstance(val, datetime.datetime) for val in val1.flat):
-> 1055     raise TypeError(
   1056         f"Input values for {self.name} class must be datetime objects"
   1057     )
   1058 if val2 is not None:

TypeError: Input values for datetime class must be datetime objects

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
File ~/Documents/codes/sunpy/sunpy/net/vso/table_response.py:105, in VSOQueryResponseTable.from_zeep_response(cls, response, client, _sort)
    103 try:
    104     # Try to use a vectorised call to parse_time
--> 105     data[col] = parse_time(data[col])
    106 except Exception:
    107     # If that fails, parse dates one by one. This is needed if
    108     # VSO returns a variety of different date format strings

File ~/Documents/codes/sunpy/sunpy/time/time.py:341, in parse_time(time_string, format, **kwargs)
    340 else:
--> 341     rt = convert_time(time_string, format=format, **kwargs)
    343 return rt

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/functools.py:909, in singledispatch.<locals>.wrapper(*args, **kw)
    906     raise TypeError(f'{funcname} requires at least '
    907                     '1 positional argument')
--> 909 return dispatch(args[0].__class__)(*args, **kw)

File ~/Documents/codes/sunpy/sunpy/time/time.py:206, in convert_time_npndarray(time_string, **kwargs)
    205 else:
--> 206     return convert_time.dispatch(object)(time_string, **kwargs)

File ~/Documents/codes/sunpy/sunpy/time/time.py:156, in convert_time(time_string, format, **kwargs)
    153 @singledispatch
    154 def convert_time(time_string, format=None, **kwargs):
    155     # default case when no type matches
--> 156     return Time(time_string, format=format, **kwargs)

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/core.py:1866, in Time.__init__(self, val, val2, format, scale, precision, in_subfmt, out_subfmt, location, copy)
   1865 else:
-> 1866     self._init_from_vals(
   1867         val, val2, format, scale, copy, precision, in_subfmt, out_subfmt
   1868     )
   1869     self.SCALES = TIME_TYPES[self.scale]

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/core.py:535, in TimeBase._init_from_vals(self, val, val2, format, scale, copy, precision, in_subfmt, out_subfmt)
    534 # Parse / convert input values into internal jd1, jd2 based on format
--> 535 self._time = self._get_time_fmt(
    536     val, val2, format, scale, precision, in_subfmt, out_subfmt
    537 )
    538 self._format = self._time.name

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/core.py:613, in TimeBase._get_time_fmt(self, val, val2, format, scale, precision, in_subfmt, out_subfmt)
    612 else:
--> 613     raise ValueError(
    614         "Input values did not match any of the formats where the format "
    615         f"keyword is optional: {problems}"
    616     ) from problems[formats[0][0]]

ValueError: Input values did not match any of the formats where the format keyword is optional: {'datetime': TypeError('Input values for datetime class must be datetime objects'), 'ymdhms': ValueError('input must be dict or table-like'), 'iso': ValueError('Time 2020-11-09 14:03: 0 does not match iso format'), 'isot': ValueError('Time 2020-11-09 14:02:38 does not match isot format'), 'yday': ValueError('Time 2020-11-09 14:02:38 does not match yday format'), 'datetime64': TypeError('Input values for datetime64 class must be datetime64 objects'), 'fits': ValueError('Time 2020-11-09 14:02:38.392 does not match fits format'), 'byear_str': ValueError('Time 2020-11-09 14:02:38.392 does not match byear_str format'), 'jyear_str': ValueError('Time 2020-11-09 14:02:38.392 does not match jyear_str format'), 'astropy_time': TypeError('Input values for astropy_time class must all be the same astropy Time type.')}

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/core.py:597, in TimeBase._get_time_fmt(self, val, val2, format, scale, precision, in_subfmt, out_subfmt)
    596 try:
--> 597     return cls(val, val2, scale, precision, in_subfmt, out_subfmt)
    598 except UnitConversionError:

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/formats.py:159, in TimeFormat.__init__(self, val1, val2, scale, precision, in_subfmt, out_subfmt, from_jd)
    158 else:
--> 159     val1, val2 = self._check_val_type(val1, val2)
    160     self.set_jds(val1, val2)

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/formats.py:1055, in TimeDatetime._check_val_type(self, val1, val2)
   1054 if not all(isinstance(val, datetime.datetime) for val in val1.flat):
-> 1055     raise TypeError(
   1056         f"Input values for {self.name} class must be datetime objects"
   1057     )
   1058 if val2 is not None:

TypeError: Input values for datetime class must be datetime objects

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
Cell In[1], line 3
      1 import astropy.time
      2 from sunpy.net import Fido, attrs as a
----> 3 Fido.search(
      4     a.Time('2020-11-09 14:00:00', end='2020-11-09 14:30:00'),
      5     a.Instrument('XRT'),
      6 )

File ~/Documents/codes/sunpy/sunpy/net/fido_factory.py:314, in UnifiedDownloaderFactory.search(self, *query)
    268 """
    269 Query for data in form of multiple parameters.
    270 
   (...)
    311 parts individually.
    312 """
    313 query = attr.and_(*query)
--> 314 results = query_walker.create(query, self)
    316 # If we have searched the VSO but no results were returned, but another
    317 # client generated results, we drop the empty VSO results for tidiness.
    318 # This is because the VSO _can_handle_query is very broad because we
    319 # don't know the full list of supported values we can search for (yet).
    320 results = [r for r in results if not isinstance(r, vso.VSOQueryResponseTable) or len(r) > 0]

File ~/Documents/codes/sunpy/sunpy/net/attr.py:613, in AttrWalker.create(self, *args, **kwargs)
    609 def create(self, *args, **kwargs):
    610     """
    611     Call the create function(s) matching the arguments to this method.
    612     """
--> 613     return self.createmm(self, *args, **kwargs)

File ~/Documents/codes/sunpy/sunpy/util/functools.py:18, in seconddispatch.<locals>.wrapper(*args, **kwargs)
     17 def wrapper(*args, **kwargs):
---> 18     return dispatcher.dispatch(args[1].__class__)(*args, **kwargs)

File ~/Documents/codes/sunpy/sunpy/net/fido_factory.py:243, in _create_and(walker, query, factory)
    241 @query_walker.add_creator(attr.AttrAnd)
    242 def _create_and(walker, query, factory):
--> 243     return factory._make_query_to_client(*query.attrs)

File ~/Documents/codes/sunpy/sunpy/net/fido_factory.py:485, in UnifiedDownloaderFactory._make_query_to_client(self, *query)
    483     if isinstance(tmpclient, vso.VSOClient):
    484         kwargs = dict(response_format="table")
--> 485     results.append(tmpclient.search(*query, **kwargs))
    487 # This method is called by `search` and the results are fed into a
    488 # UnifiedResponse object.
    489 return results

File ~/Documents/codes/sunpy/sunpy/net/vso/vso.py:249, in VSOClient.search(self, response_format, *query)
    247     response = QueryResponse.create(responses)
    248 else:
--> 249     response = VSOQueryResponseTable.from_zeep_response(responses, client=self)
    251 for ex in exceptions:
    252     response.add_error(ex)

File ~/Documents/codes/sunpy/sunpy/net/vso/table_response.py:113, in VSOQueryResponseTable.from_zeep_response(cls, response, client, _sort)
    111 for i, t in enumerate(data[col]):
    112     if t is not None:
--> 113         times.append(parse_time(t))
    114     else:
    115         # Create a dummy time and mask it later
    116         times.append(Time(val=0, format='mjd'))

File ~/Documents/codes/sunpy/sunpy/time/time.py:341, in parse_time(time_string, format, **kwargs)
    339     rt = Time.now()
    340 else:
--> 341     rt = convert_time(time_string, format=format, **kwargs)
    343 return rt

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/functools.py:909, in singledispatch.<locals>.wrapper(*args, **kw)
    905 if not args:
    906     raise TypeError(f'{funcname} requires at least '
    907                     '1 positional argument')
--> 909 return dispatch(args[0].__class__)(*args, **kw)

File ~/Documents/codes/sunpy/sunpy/time/time.py:248, in convert_time_str(time_string, **kwargs)
    245         pass
    247 # when no format matches, call default function
--> 248 return convert_time.dispatch(object)(time_string, **kwargs)

File ~/Documents/codes/sunpy/sunpy/time/time.py:156, in convert_time(time_string, format, **kwargs)
    153 @singledispatch
    154 def convert_time(time_string, format=None, **kwargs):
    155     # default case when no type matches
--> 156     return Time(time_string, format=format, **kwargs)

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/core.py:1866, in Time.__init__(self, val, val2, format, scale, precision, in_subfmt, out_subfmt, location, copy)
   1864         self._set_scale(scale)
   1865 else:
-> 1866     self._init_from_vals(
   1867         val, val2, format, scale, copy, precision, in_subfmt, out_subfmt
   1868     )
   1869     self.SCALES = TIME_TYPES[self.scale]
   1871 if self.location is not None and (
   1872     self.location.size > 1 and self.location.shape != self.shape
   1873 ):

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/core.py:535, in TimeBase._init_from_vals(self, val, val2, format, scale, copy, precision, in_subfmt, out_subfmt)
    532 mask, val, val2 = _check_for_masked_and_fill(val, val2)
    534 # Parse / convert input values into internal jd1, jd2 based on format
--> 535 self._time = self._get_time_fmt(
    536     val, val2, format, scale, precision, in_subfmt, out_subfmt
    537 )
    538 self._format = self._time.name
    540 # Hack from #9969 to allow passing the location value that has been
    541 # collected by the TimeAstropyTime format class up to the Time level.
    542 # TODO: find a nicer way.

File ~/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/astropy/time/core.py:613, in TimeBase._get_time_fmt(self, val, val2, format, scale, precision, in_subfmt, out_subfmt)
    611             problems[name] = err
    612 else:
--> 613     raise ValueError(
    614         "Input values did not match any of the formats where the format "
    615         f"keyword is optional: {problems}"
    616     ) from problems[formats[0][0]]

ValueError: Input values did not match any of the formats where the format keyword is optional: {'datetime': TypeError('Input values for datetime class must be datetime objects'), 'ymdhms': ValueError('input must be dict or table-like'), 'iso': ValueError('Time 2020-11-09 14:03: 0 does not match iso format'), 'isot': ValueError('Time 2020-11-09 14:03: 0 does not match isot format'), 'yday': ValueError('Time 2020-11-09 14:03: 0 does not match yday format'), 'datetime64': TypeError('Input values for datetime64 class must be datetime64 objects'), 'fits': ValueError('Time 2020-11-09 14:03: 0.172 does not match fits format'), 'byear_str': ValueError('Time 2020-11-09 14:03: 0.172 does not match byear_str format'), 'jyear_str': ValueError('Time 2020-11-09 14:03: 0.172 does not match jyear_str format'), 'astropy_time': TypeError('Input values for astropy_time class must all be the same astropy Time type.')}

Additionally, here's the body of the XML response from the VSO,

 <soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:VSO="http://virtualsolar.org/VSO/VSOi">
  <soap-env:Body>
    <VSO:Query>
      <body>
        <block>
          <time>
            <start>20201109140000</start>
            <end>20201109143000</end>
          </time>
          <instrument>XRT</instrument>
        </block>
      </body>
    </VSO:Query>
  </soap-env:Body>
</soap-env:Envelope>
 [sunpy.net.vso.zeep_plugins]
DEBUG: VSO Response:
 <soap:Envelope xmlns:VSO="http://virtualsolar.org/VSO/VSOi" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soap:Body>
    <VSO:QueryResponse>
      <body>
        <provideritem>
          <record>
            <recorditem>
              <wave>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
                <wavemax>220</wavemax>
              </wave>
              <size>607680</size>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140238.3.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140238.3.fits%3C/fileid)>
              <physobs>Intensity</physobs>
              <info/>
              <extent>
                <type>PARTIAL_SUN</type>
                <x>54.004700</x>
                <y>-454.092255</y>
              </extent>
              <source>Hinode</source>
              <time>
                <end>2020-11-09 14:02:39.415</end>
                <start>2020-11-09 14:02:38.392</start>
              </time>
              <instrument>XRT</instrument>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <physobs>Intensity</physobs>
              <info/>
              <extent>
                <y>-454.021790</y>
                <x>54.120800</x>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <time>
                <start>2020-11-09 14:02:41.556</start>
                <end>2020-11-09 14:02:41.685</end>
              </time>
              <instrument>XRT</instrument>
              <wave>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
                <waveunit>Angstrom</waveunit>
                <wavemax>220</wavemax>
              </wave>
              <size>607680</size>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140241.5.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140241.5.fits%3C/fileid)>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <extent>
                <x>54.059300</x>
                <y>-454.082214</y>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <time>
                <end>2020-11-09 14:02:51.885</end>
                <start>2020-11-09 14:02:46.089</start>
              </time>
              <instrument>XRT</instrument>
              <physobs>Intensity</physobs>
              <info/>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140246.0.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140246.0.fits%3C/fileid)>
              <size>607680</size>
              <wave>
                <wavemax>30</wavemax>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
              </wave>
            </recorditem>
            <recorditem>
              <info/>
              <physobs>Intensity</physobs>
              <extent>
                <y>-454.069153</y>
                <x>53.870800</x>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <time>
                <start>2020-11-09 14:03: 0.172</start>
                <end>2020-11-09 14:03: 0.895</end>
              </time>
              <instrument>XRT</instrument>
              <size>607680</size>
              <wave>
                <wavemax>30</wavemax>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
              </wave>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140300.1.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140300.1.fits%3C/fileid)>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <info/>
              <physobs>Intensity</physobs>
              <extent>
                <x>53.808800</x>
                <y>-454.190399</y>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <time>
                <start>2020-11-09 14:03: 5.094</start>
                <end>2020-11-09 14:03: 5.276</end>
              </time>
              <instrument>XRT</instrument>
              <size>607680</size>
              <wave>
                <wavemax>220</wavemax>
                <waveunit>Angstrom</waveunit>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
              </wave>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140305.0.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140305.0.fits%3C/fileid)>
            </recorditem>
            <recorditem>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140309.5.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140309.5.fits%3C/fileid)>
              <size>607680</size>
              <wave>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
                <wavemax>30</wavemax>
              </wave>
              <extent>
                <type>PARTIAL_SUN</type>
                <x>53.871500</x>
                <y>-454.007446</y>
              </extent>
              <source>Hinode</source>
              <instrument>XRT</instrument>
              <time>
                <start>2020-11-09 14:03: 9.588</start>
                <end>2020-11-09 14:03:10.312</end>
              </time>
              <physobs>Intensity</physobs>
              <info/>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <extent>
                <type>PARTIAL_SUN</type>
                <y>-454.006470</y>
                <x>54.459000</x>
              </extent>
              <source>Hinode</source>
              <time>
                <start>2020-11-09 14:08: 0.663</start>
                <end>2020-11-09 14:08: 0.844</end>
              </time>
              <instrument>XRT</instrument>
              <info/>
              <physobs>Intensity</physobs>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140800.6.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140800.6.fits%3C/fileid)>
              <size>607680</size>
              <wave>
                <wavemax>220</wavemax>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
              </wave>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140805.1.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_140805.1.fits%3C/fileid)>
              <size>607680</size>
              <wave>
                <wavemax>30</wavemax>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
                <waveunit>Angstrom</waveunit>
              </wave>
              <extent>
                <y>-454.190216</y>
                <x>54.396300</x>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <time>
                <end>2020-11-09 14:08: 6.219</end>
                <start>2020-11-09 14:08: 5.194</start>
              </time>
              <instrument>XRT</instrument>
              <physobs>Intensity</physobs>
              <info/>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <info/>
              <physobs>Intensity</physobs>
              <extent>
                <y>-454.466492</y>
                <x>54.878900</x>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <instrument>XRT</instrument>
              <time>
                <end>2020-11-09 14:13: 0.943</end>
                <start>2020-11-09 14:13: 0.762</start>
              </time>
              <wave>
                <waveunit>Angstrom</waveunit>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
                <wavemax>220</wavemax>
              </wave>
              <size>607680</size>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_141300.7.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_141300.7.fits%3C/fileid)>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <wave>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
                <wavemax>30</wavemax>
              </wave>
              <size>607680</size>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_141305.2.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_141305.2.fits%3C/fileid)>
              <info/>
              <physobs>Intensity</physobs>
              <source>Hinode</source>
              <extent>
                <type>PARTIAL_SUN</type>
                <x>55.066400</x>
                <y>-454.468323</y>
              </extent>
              <time>
                <start>2020-11-09 14:13: 5.293</start>
                <end>2020-11-09 14:13: 6.318</end>
              </time>
              <instrument>XRT</instrument>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <time>
                <end>2020-11-09 14:18: 1.042</end>
                <start>2020-11-09 14:18: 0.861</start>
              </time>
              <instrument>XRT</instrument>
              <extent>
                <y>-455.339813</y>
                <x>56.122600</x>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <info/>
              <physobs>Intensity</physobs>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_141800.8.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_141800.8.fits%3C/fileid)>
              <size>607680</size>
              <wave>
                <wavemax>220</wavemax>
                <waveunit>Angstrom</waveunit>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
              </wave>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <wave>
                <wavemax>30</wavemax>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
                <waveunit>Angstrom</waveunit>
              </wave>
              <size>607680</size>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_141805.3.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_141805.3.fits%3C/fileid)>
              <physobs>Intensity</physobs>
              <info/>
              <extent>
                <y>-455.216827</y>
                <x>56.062700</x>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <time>
                <start>2020-11-09 14:18: 5.384</start>
                <end>2020-11-09 14:18: 6.409</end>
              </time>
              <instrument>XRT</instrument>
            </recorditem>
            <recorditem>
              <extent>
                <x>41.243700</x>
                <y>-448.600616</y>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <instrument>XRT</instrument>
              <time>
                <start>2020-11-09 14:23: 5.458</start>
                <end>2020-11-09 14:23: 5.459</end>
              </time>
              <physobs>Intensity</physobs>
              <info/>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142305.4.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142305.4.fits%3C/fileid)>
              <wave>
                <wavemin>4118</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
                <wavemax>4496</wavemax>
              </wave>
              <size>607680</size>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <wave>
                <wavemax>4496</wavemax>
                <wavemin>4118</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
              </wave>
              <size>607680</size>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142307.9.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142307.9.fits%3C/fileid)>
              <physobs>Intensity</physobs>
              <info/>
              <time>
                <start>2020-11-09 14:23: 7.946</start>
                <end>2020-11-09 14:23: 7.947</end>
              </time>
              <instrument>XRT</instrument>
              <source>Hinode</source>
              <extent>
                <type>PARTIAL_SUN</type>
                <y>-455.738495</y>
                <x>57.101000</x>
              </extent>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <size>607680</size>
              <wave>
                <wavemax>220</wavemax>
                <waveunit>Angstrom</waveunit>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
              </wave>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142334.9.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142334.9.fits%3C/fileid)>
              <info/>
              <physobs>Intensity</physobs>
              <instrument>XRT</instrument>
              <time>
                <end>2020-11-09 14:23:35.996</end>
                <start>2020-11-09 14:23:34.972</start>
              </time>
              <extent>
                <type>PARTIAL_SUN</type>
                <y>-455.769135</y>
                <x>56.927300</x>
              </extent>
              <source>Hinode</source>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142338.4.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142338.4.fits%3C/fileid)>
              <wave>
                <wavemax>220</wavemax>
                <waveunit>Angstrom</waveunit>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
              </wave>
              <size>607680</size>
              <instrument>XRT</instrument>
              <time>
                <start>2020-11-09 14:23:38.473</start>
                <end>2020-11-09 14:23:38.602</end>
              </time>
              <extent>
                <type>PARTIAL_SUN</type>
                <y>-455.710175</y>
                <x>57.171800</x>
              </extent>
              <source>Hinode</source>
              <physobs>Intensity</physobs>
              <info/>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <instrument>XRT</instrument>
              <time>
                <end>2020-11-09 14:23:48.794</end>
                <start>2020-11-09 14:23:42.997</start>
              </time>
              <extent>
                <x>56.992700</x>
                <y>-455.785950</y>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <physobs>Intensity</physobs>
              <info/>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142342.9.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142342.9.fits%3C/fileid)>
              <size>607680</size>
              <wave>
                <wavemax>30</wavemax>
                <waveunit>Angstrom</waveunit>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
              </wave>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <size>607680</size>
              <wave>
                <wavemax>30</wavemax>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
              </wave>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142350.9.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142350.9.fits%3C/fileid)>
              <physobs>Intensity</physobs>
              <info/>
              <instrument>XRT</instrument>
              <time>
                <start>2020-11-09 14:23:50.978</start>
                <end>2020-11-09 14:23:52.003</end>
              </time>
              <extent>
                <type>PARTIAL_SUN</type>
                <y>-455.543182</y>
                <x>57.116600</x>
              </extent>
              <source>Hinode</source>
            </recorditem>
            <recorditem>
              <info/>
              <physobs>Intensity</physobs>
              <extent>
                <type>PARTIAL_SUN</type>
                <y>-455.727081</y>
                <x>57.176200</x>
              </extent>
              <source>Hinode</source>
              <instrument>XRT</instrument>
              <time>
                <start>2020-11-09 14:23:56.510</start>
                <end>2020-11-09 14:23:56.692</end>
              </time>
              <wave>
                <waveunit>Angstrom</waveunit>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
                <wavemax>220</wavemax>
              </wave>
              <size>607680</size>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142356.5.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142356.5.fits%3C/fileid)>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <provider>SAO</provider>
              <size>607680</size>
              <wave>
                <wavemax>30</wavemax>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
                <waveunit>Angstrom</waveunit>
              </wave>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142401.0.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142401.0.fits%3C/fileid)>
              <physobs>Intensity</physobs>
              <info/>
              <extent>
                <x>56.998300</x>
                <y>-455.619568</y>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <time>
                <end>2020-11-09 14:24: 2.026</end>
                <start>2020-11-09 14:24: 1.002</start>
              </time>
              <instrument>XRT</instrument>
            </recorditem>
            <recorditem>
              <size>607680</size>
              <wave>
                <wavemax>220</wavemax>
                <wavemin>3</wavemin>
                <wavetype>Broad</wavetype>
                <waveunit>Angstrom</waveunit>
              </wave>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142853.0.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142853.0.fits%3C/fileid)>
              <info/>
              <physobs>Intensity</physobs>
              <time>
                <end>2020-11-09 14:28:53.258</end>
                <start>2020-11-09 14:28:53.077</start>
              </time>
              <instrument>XRT</instrument>
              <extent>
                <y>-456.002899</y>
                <x>58.232700</x>
                <type>PARTIAL_SUN</type>
              </extent>
              <source>Hinode</source>
              <provider>SAO</provider>
            </recorditem>
            <recorditem>
              <wave>
                <waveunit>Angstrom</waveunit>
                <wavetype>Broad</wavetype>
                <wavemin>3</wavemin>
                <wavemax>30</wavemax>
              </wave>
              <size>607680</size>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142857.6.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1400/L1_XRT20201109_142857.6.fits%3C/fileid)>
              <physobs>Intensity</physobs>
              <info/>
              <time>
                <end>2020-11-09 14:28:58.627</end>
                <start>2020-11-09 14:28:57.602</start>
              </time>
              <instrument>XRT</instrument>
              <source>Hinode</source>
              <extent>
                <type>PARTIAL_SUN</type>
                <x>58.171300</x>
                <y>-456.063324</y>
              </extent>
              <provider>SAO</provider>
            </recorditem>
          </record>
          <version>1</version>
          <no_of_records_found>22</no_of_records_found>
          <no_of_records_returned>22</no_of_records_returned>
          <provider>SAO</provider>
        </provideritem>
      </body>
    </VSO:QueryResponse>
  </soap:Body>
</soap:Envelope>

Screenshots

When searching through the VSO web form, this returns results:

image

System Details

============================== sunpy Installation Information

General ####### OS: Mac OS 12.6.6 Arch: 64bit, (arm) sunpy: 5.1.dev259+g52b1428e9 Installation path: /Users/wtbarnes/mambaforge/envs/sunpy-dev/lib/python3.11/site-packages/sunpy-5.1.dev259+g52b1428e9.dist-info

Required Dependencies ##################### astropy: 5.3.4 numpy: 1.26.1 packaging: 23.2 parfive: 2.0.2

Optional Dependencies ##################### asdf: 3.0.0 asdf-astropy: 0.4.0 beautifulsoup4: 4.12.2 cdflib: 1.2.2 dask: 2023.10.0 drms: 0.6.4 glymur: 0.12.8 h5netcdf: 1.2.0 h5py: 3.10.0 lxml: 4.9.3 matplotlib: 3.8.0 mpl-animators: 1.1.0 pandas: 2.1.1 python-dateutil: 2.8.2 reproject: 0.12.0 scikit-image: 0.22.0 scipy: 1.11.3 sqlalchemy: 2.0.22 tqdm: 4.66.1 zeep: 4.2.1

Installation method

No response

AlisdairDavey commented 11 months ago

We have transitioned to returning time with millisecond precision where that information is available. The issue was in my $VSOtime = sprintf "%.4d-%.2d-%.2d %.2d:%.2d:%6.3f" which does not pad the float resulting in a blank space in the time string for seconds < 10. Fixed it and results are now correct. 


'time' => {
   'start' => '2020-11-09 14:03:00.172'
   'end' => '2020-11-09 14:03:00.895',                    
}
wtbarnes commented 11 months ago

Thanks @AlisdairDavey! My above query is now working.