BUG - End date not honored by CLI/Python API

mfacchinelli commented 2 weeks ago

Description of the issue

When using the CLI/Python APIs to request data for a specific time period, the end date does not seem to be respected.

Steps to reproduce the issue

In a shell terminal (or Python) run:

imap-data-access query --instrument mag --start-date 20240314 --end-date 20240315

This returns:

Found [14] matching files
-----------------------------------------------------------------------------------------------------------------|
Instrument|Data Level|Descriptor|Start Date|Repointing|Version|Filename                                          |
-----------------------------------------------------------------------------------------------------------------|
mag       |l0        |raw       |20240508  |          |v001   |imap_mag_l0_raw_20240508_v001.pkts                |
mag       |l0        |raw       |20240508  |          |v002   |imap_mag_l0_raw_20240508_v002.pkts                |
mag       |l0        |raw       |20240508  |          |v003   |imap_mag_l0_raw_20240508_v003.pkts                |
mag       |l1a       |norm-raw  |20240508  |          |v003   |imap_mag_l1a_norm-raw_20240508_v003.cdf           |
mag       |l1a       |norm-mago |20240508  |          |v003   |imap_mag_l1a_norm-mago_20240508_v003.cdf          |
mag       |l1a       |norm-magi |20240508  |          |v003   |imap_mag_l1a_norm-magi_20240508_v003.cdf          |
mag       |l0        |raw       |20240508  |          |v004   |imap_mag_l0_raw_20240508_v004.pkts                |
mag       |l1a       |norm-raw  |20240508  |          |v004   |imap_mag_l1a_norm-raw_20240508_v004.cdf           |
mag       |l1a       |norm-mago |20240508  |          |v004   |imap_mag_l1a_norm-mago_20240508_v004.cdf          |
mag       |l1a       |norm-magi |20240508  |          |v004   |imap_mag_l1a_norm-magi_20240508_v004.cdf          |
mag       |l0        |raw       |20240508  |          |v005   |imap_mag_l0_raw_20240508_v005.pkts                |
mag       |l0        |raw       |20240508  |          |v000   |imap_mag_l0_raw_20240508_v000.pkts                |
mag       |l0        |raw       |20240509  |          |v000   |imap_mag_l0_raw_20240509_v000.pkts                |
mag       |l0        |raw       |20240314  |          |v000   |imap_mag_l0_raw_20240314_v000.pkts                |
-----------------------------------------------------------------------------------------------------------------|

which includes dates well past the cutoff.

Expected behavior (What should happen)

With the web APIs:

https://api.dev.imap-mission.com/query?instrument=mag&start_date=20240314&end_date=20240315

the end date is honored:

[
  {
    "file_path": "imap/mag/l0/2024/03/imap_mag_l0_raw_20240314_v000.pkts",
    "instrument": "mag",
    "data_level": "l0",
    "descriptor": "raw",
    "start_date": "20240314",
    "repointing": null,
    "version": "v000",
    "extension": "pkts",
    "ingestion_date": "2024-06-14 14:37:20"
  }
]

Actual behavior (What does happen)

See reproduction steps.

Code Snippet:

imap-data-access query --instrument mag --start-date 20240314 --end-date 20240315

Additional notes, affected areas, and suggested fixes

No response

greglucas commented 2 weeks ago

I am guessing that we need to add end_date to this list: https://github.com/IMAP-Science-Operations-Center/imap-data-access/blob/bde27e7b9c56fe5b2b613afa1f588eb28e0589fb/imap_data_access/cli.py#L96

If you add verbose logging you can also see that we aren't passing it to the API.

imap-data-access -v query --instrument mag --start-date 20240314 --end-date 20240315
INFO:imap_data_access.io:Querying data archive for {'instrument': 'mag', 'start_date': '20240314'} with url https://api.dev.imap-mission.com/query?instrument=mag&start_date=20240314

mfacchinelli commented 1 week ago

Thanks @greglucas, that was helpful! I've created #48 for the fix

IMAP-Science-Operations-Center / imap-data-access