search

sunpy / sunpy

SunPy - Python for Solar Physics
http://www.sunpy.org
BSD 2-Clause "Simplified" License
903 stars 582 forks source link

How to search for VSO EXTENT_TYPE? #4048

Open dstansby opened 4 years ago

dstansby commented 4 years ago

I'm trying to use Fido to search the vso by the EVENT_TYPE keyword, but I'm not entirely sure what attr I should be using for this. Sepcifically I want to search for EXTENT_TYPE = 'synotpic'.

The most obvious one seems to be sunpy.net.vso.attrs.Extent, but this requires x, y, width, length values which I don't want to specify. Does anyone have any idea?

abhijeetmanhas commented 4 years ago 
class Extent(_attr.DataAttr):
    """
    Specify the spatial field-of-view of the query. Due to a bug in the VSO,
    the Extent attribute is not used.

    """

In the docstring for vso attrs file, it is written their is bug in VSO. May I know what's the bug and whether it is fixed now?

dstansby commented 4 years ago

Looks like @wafels added that comment a few years back, perhaps he remembers?

ejm4567 commented 1 year ago

I think the issue with Extent in the VSO is that not all Data Providers support searches against it, and if they do, only the Extent[type] attribute. The VSO never implemented searches against the FOV dimensions, or the co-ordinates of the reference pixel. We are working on implementing a full search of all the Extent variables for Data Providers that have that data available.

Cadair commented 1 year ago

Looks like we should maybe make a new attr ExtentType to support these queries?

dstansby commented 1 year ago

For refernce sunpy now has an ExtentType attr that is used to search in the GONGClient, the work to fix this issue is presumably make the VSOClient understand that attr and search with it.

wtbarnes commented 9 months ago

Well I've once again run into this issue and was very confused by the fact that we have an ExtentType attr, the VSO returns an ExtentType in the response, yet I'm not able to search the VSO using the ExtentType attr!

As an example, the following fails,

from sunpy.net import Fido, attrs as a
q = Fido.search(
    a.Time('2020-11-09T18:19:00', '2020-11-09T18:28:00'),
    a.Instrument('XRT'),
    a.ExtentType('FULLDISK')
)

with

---------------------------------------------------------------------------
NoMatchError                              Traceback (most recent call last)
Input In [32], in <cell line: 1>()
----> 1 q = Fido.search(
      2     a.Time('2020-11-09T18:19:00', '2020-11-09T18:28:00'), #near='2020-11-09T18:19:00'),
      3     a.Instrument('XRT'),
      4     a.ExtentType('FULLDISK')
      5 )

File ~/mambaforge/envs/mocksipipeline/lib/python3.9/site-packages/sunpy/net/fido_factory.py:315, in UnifiedDownloaderFactory.search(self, *query)
    269 """
    270 Query for data in form of multiple parameters.
    271 
   (...)
    312 parts individually.
    313 """
    314 query = attr.and_(*query)
--> 315 results = query_walker.create(query, self)
    317 # If we have searched the VSO but no results were returned, but another
    318 # client generated results, we drop the empty VSO results for tidiness.
    319 # This is because the VSO _can_handle_query is very broad because we
    320 # don't know the full list of supported values we can search for (yet).
    321 results = [r for r in results if not isinstance(r, vso.VSOQueryResponseTable) or len(r) > 0]

File ~/mambaforge/envs/mocksipipeline/lib/python3.9/site-packages/sunpy/net/attr.py:613, in AttrWalker.create(self, *args, **kwargs)
    609 def create(self, *args, **kwargs):
    610     """
    611     Call the create function(s) matching the arguments to this method.
    612     """
--> 613     return self.createmm(self, *args, **kwargs)

File ~/mambaforge/envs/mocksipipeline/lib/python3.9/site-packages/sunpy/util/functools.py:18, in seconddispatch.<locals>.wrapper(*args, **kwargs)
     17 def wrapper(*args, **kwargs):
---> 18     return dispatcher.dispatch(args[1].__class__)(*args, **kwargs)

File ~/mambaforge/envs/mocksipipeline/lib/python3.9/site-packages/sunpy/net/fido_factory.py:244, in _create_and(walker, query, factory)
    242 @query_walker.add_creator(attr.AttrAnd)
    243 def _create_and(walker, query, factory):
--> 244     return factory._make_query_to_client(*query.attrs)

File ~/mambaforge/envs/mocksipipeline/lib/python3.9/site-packages/sunpy/net/fido_factory.py:481, in UnifiedDownloaderFactory._make_query_to_client(self, *query)
    467 def _make_query_to_client(self, *query):
    468     """
    469     Given a query, look up the client and perform the query.
    470 
   (...)
    479         Instance of client class
    480     """
--> 481     candidate_widget_types = self._check_registered_widgets(*query)
    482     results = []
    483     for client in candidate_widget_types:

File ~/mambaforge/envs/mocksipipeline/lib/python3.9/site-packages/sunpy/net/fido_factory.py:463, in UnifiedDownloaderFactory._check_registered_widgets(self, *args)
    460 n_matches = len(candidate_widget_types)
    461 if n_matches == 0:
    462     # There is no default client
--> 463     raise NoMatchError("This query was not understood by any clients. Did you miss an OR?")
    465 return candidate_widget_types

NoMatchError: This query was not understood by any clients. Did you miss an OR?

while the same query without an ExtentType succeds and returns a response containing columns with "Extent Type",

'Results from 1 Provider:

7 Results from the VSOClient:

VSOQueryResponseTable length=7\n\n\n\n\n\n\n\n\n\n\n\n
Start TimeEnd TimeSourceInstrumentWavelengthProviderPhysobsWavetypeExtent XExtent YExtent TypeSize
AngstromMibyte
TimeTimestr6str3float64[2]str3str9str5str10str9str11float64
2020-11-09 18:19:02.6512020-11-09 18:19:25.793HinodeXRT3.0 .. 30.0SAOIntensityBroad-16.936184-6.063051FULLDISK4114.6875
2020-11-09 18:19:37.2772020-11-09 18:19:37.278HinodeXRT4118.0 .. 4496.0SAOIntensityBroad-33.4757351.652862FULLDISK16402.5
2020-11-09 18:19:46.6482020-11-09 18:19:46.649HinodeXRT4118.0 .. 4496.0SAOIntensityBroad-17.441919-5.631000FULLDISK16402.5
2020-11-09 18:27:38.8162020-11-09 18:27:38.817HinodeXRT4118.0 .. 4496.0SAOIntensityBroad-32.1874352.779237PARTIAL_SUN1043.4375
2020-11-09 18:27:41.8032020-11-09 18:27:41.804HinodeXRT4118.0 .. 4496.0SAOIntensityBroad-16.024746-4.506716PARTIAL_SUN1043.4375
2020-11-09 18:27:50.5072020-11-09 18:27:51.531HinodeXRT3.0 .. 30.0SAOIntensityBroad-16.207014-4.565342FULLDISK1043.4375
2020-11-09 18:27:54.3252020-11-09 18:27:55.770HinodeXRT3.0 .. 30.0SAOIntensityBroad-16.144033-4.321879FULLDISK1043.4375

'

wtbarnes commented 9 months ago

OK, some more progress. Doing the following,

import sunpy
sunpy.log.setLevel('DEBUG')
from sunpy.net import Fido, attrs as a
import astropy.units as u
q = Fido.search(
    a.Time('2020-11-09T18:19:45', '2020-11-09T18:27:40'), #near='2020-11-09T18:19:00'),
    a.Instrument('XRT'),
)

succeeds and returns two results (as expected) and I get the following debug output,

DEBUG: VSO Request:
 <soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:VSO="http://virtualsolar.org/VSO/VSOi">
  <soap-env:Body>
    <VSO:Query>
      <body>
        <block>
          <time>
            <start>20201109181945</start>
            <end>20201109182740</end>
          </time>
          <instrument>XRT</instrument>
        </block>
      </body>
    </VSO:Query>
  </soap-env:Body>
</soap-env:Envelope>
 [sunpy.net.vso.zeep_plugins]
DEBUG: VSO Response:
 <soap:Envelope xmlns:VSO="http://virtualsolar.org/VSO/VSOi" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soap:Body>
    <VSO:QueryResponse>
      <body>
        <provideritem>
          <version>1</version>
          <record>
            <recorditem>
              <time>
                <start>2020-11-09 18:19:46.648</start>
                <end>2020-11-09 18:19:46.649</end>
              </time>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1800/L1_XRT20201109_181946.6.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1800/L1_XRT20201109_181946.6.fits%3C/fileid)>
              <instrument>XRT</instrument>
              <wave>
                <wavemax>4496</wavemax>
                <wavetype>Broad</wavetype>
                <wavemin>4118</wavemin>
                <waveunit>Angstrom</waveunit>
              </wave>
              <provider>SAO</provider>
              <extent>
                <y>-5.631000</y>
                <x>-17.441919</x>
                <type>FULLDISK</type>
              </extent>
              <physobs>Intensity</physobs>
              <info/>
              <size>16796160</size>
              <source>Hinode</source>
            </recorditem>
            <recorditem>
              <wave>
                <wavemax>4496</wavemax>
                <wavetype>Broad</wavetype>
                <wavemin>4118</wavemin>
                <waveunit>Angstrom</waveunit>
              </wave>
              <instrument>XRT</instrument>
              <extent>
                <x>-32.187435</x>
                <y>2.779237</y>
                <type>PARTIAL_SUN</type>
              </extent>
              <provider>SAO</provider>
              <time>
                <start>2020-11-09 18:27:38.816</start>
                <end>2020-11-09 18:27:38.817</end>
              </time>
              <fileid>[https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1800/L1_XRT20201109_182738.8.fits</fileid](https://umbra.nascom.nasa.gov/hinode/xrt/level1/2020/11/09/H1800/L1_XRT20201109_182738.8.fits%3C/fileid)>
              <source>Hinode</source>
              <size>1068480</size>
              <physobs>Intensity</physobs>
              <info/>
            </recorditem>
          </record>
          <no_of_records_returned>2</no_of_records_returned>
          <provider>SAO</provider>
          <no_of_records_found>2</no_of_records_found>
        </provideritem>
      </body>
    </VSO:QueryResponse>
  </soap:Body>
</soap:Envelope>
 [sunpy.net.vso.zeep_plugins]

However, when I try to add an extent type using Extent, I don't get an exception because the VSOClient does understand the Extent attr, but I get 0 results,

q = Fido.search(
    a.Time('2020-11-09T18:19:45', '2020-11-09T18:27:40'), #near='2020-11-09T18:19:00'),
    a.Instrument('XRT'),
    a.vso.Extent(None, None, None, None, 'FULLDISK'),
)

with the following debug output,

DEBUG: VSO Request:
 <soap-env:Envelope xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:VSO="http://virtualsolar.org/VSO/VSOi">
  <soap-env:Body>
    <VSO:Query>
      <body>
        <block>
          <time>
            <start>20201109181945</start>
            <end>20201109182740</end>
          </time>
          <instrument>XRT</instrument>
          <extent>
            <type>FULLDISK</type>
          </extent>
        </block>
      </body>
    </VSO:Query>
  </soap-env:Body>
</soap-env:Envelope>
 [sunpy.net.vso.zeep_plugins]
DEBUG: VSO Response:
 <soap:Envelope xmlns:VSO="http://virtualsolar.org/VSO/VSOi" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <soap:Body>
    <VSO:QueryResponse>
      <body>
        <provideritem>
          <no_of_records_returned>0</no_of_records_returned>
          <no_of_records_found>0</no_of_records_found>
          <version>1</version>
          <record/>
          <provider>SAO</provider>
        </provideritem>
      </body>
    </VSO:QueryResponse>
  </soap:Body>
</soap:Envelope>
 [sunpy.net.vso.zeep_plugins]

@ejm4567 per your above comment, is this because the XRT data provider does not understand EXTENT_TYPE?

AlisdairDavey commented 9 months ago

As the author of the XRT data provider cough I believe it should do. I think I made tests that use the extent type. I'll check tomorrow and make sure this is the case!!

wtbarnes commented 7 months ago

@AlisdairDavey Just checking in to see if you found out anything more regarding the use of EXTENT_TYPE with XRT? I'm still finding that this is returning 0 results.

AlisdairDavey commented 7 months ago

@wtbarnes Crap! Sorry - now in large letters on my board to do!!! Get back to work next week after my knee replacement and will work on it then.