search

irfu / irfu-matlab

Matlab routines to work with space data, particularly with MMS and Cluster/CAA data. Also some general plasma routines.
56 stars 46 forks source link

ESA Cluster is changing its CLI access & download system from "CAIO" to "TAP". #75

Closed thomas-nilsson-irfu closed 2 years ago

thomas-nilsson-irfu commented 3 years ago

ESA Cluster is about to change its command line access used for downloading Cluster data.

Adding an issue here to keep track of this, I just got an e-mail from Per-Arne with some of the following information (part of it from SOWG 74 and 75).

CAIO (Cluster Science Archive Inter-Operability Subsystem) is changing to 'New' command line or TAP (Table Access Protocol). Once the beta is successful, full functionality (up to old system) available, and help pages complete, an email will go out to all CSA members, advising of the change. After this official launch, the two systems will run in parallel for ~6 months.

This has some implications for our irfu-matlab scripts which download Cluster metadata and actual data. I'm thinking primarily of files like caa_download.m and perhaps also caa_meta.m. But I'm not sure if it is used elsewhere as well...

Some of the differences are briefly described here: https://www.cosmos.esa.int/web/csa/caiototap#MatLab_dl

It is a bit of an open question who will re-write this, but nevertheless I'm adding it here for reference for now.

thomas-nilsson-irfu commented 3 years ago

Just adding a note (sort of progress update) of what works so far and what does not.

Works (to be tested with old code to see if output is identical) First of all, for all of the values below:

irf;
tint = irf.tint('2005-01-01T05:00:00.000Z/2005-01-01T05:10:00.000Z');
dataSet = 'C3_CP_FGM_5VPS';

Does not work (in our code):

Edited 2021-06-14T17:09 CEST: (tint, 'list:C3_CP_FGM*') now works (following commit: 0d188ae2fe5a4c4f7273c8e3053d2e2855e1a792). Also while testing the old code it appear it did not work in the old interface for plain ('list') or ('listdesc') nor ('tint', 'inventory') arguments (but they are tested successfully with TAP. as per above).

Edited 2021-06-15T10:00 CEST: I did a quick grep for all function calls to caa_download.m in our irfu-matlab code, (i.e. grep -nrE "^[[:blank:]]*[^%].*caa_download\(.*\)" *), and sorted them based on what parameters each used and found some more test cases which have now been updated above (only one test case per unique combination of parameter types). Some of them are not yet expected to work as Cluster's TAP interface not yet have the necessary support.

Edited 2021-07-29T11:00 CEST: ESA Cluster TAP has now added support for Asynchronous and Streaming data requests (tested with argument 'schedule' & 'stream' respectively). However the new interface does not yet, according to the website, support Metadata header requests (which is what I suspect caa_meta.m is relying upon).

Edited 2021-10-14T14:16 CEST: ESA Cluster TAP is now live, I did a quick re-test of the different arguments used above and some appears to no longer work (despite them working before). Something may however have changed on TAP at ESA backend since the last time these arguments were tested. Updated 2021-10-14T16:42 CEST: Now all appears fixed again, including meta_data('create'); although this specific command takes a long time to complete.

thomas-nilsson-irfu commented 2 years ago

Just a general update related to this issue:

Today the new Cluster CLI was announced as "live", meaning the 6 months transition time began ticking, after which the old access method will be retired.

Quoting the e-mail from ESA:

The Cluster Science Archive (CSA) is proud to present Release 3.0, in which we have upgraded the command line access from CAIO to TAP.

Table Access Protocol (TAP) uses Astronomical Data Query Language (ADQL), an expanded version of SQL (Structured Query Language), to access the contents of the CSA via scripts or the command line, both for data and metadata requests.

Especially for metadata requests, TAP enables much more powerful searches to be launched for the information you need.

This change requires some changes to any scripts you may have to access data or metadata from the CSA.

Help to adapt is included in new user manual/help pages, which can be found at: https://www.cosmos.esa.int/web/csa-guide/home where there is also a link to a quick-start guide to adapting your old scripts to use TAP.

Both systems will be in place for 6 months; after this, the CAIO system will be retired, so please make your scripts are ready.

So the special branch for this issue should very soon have to be merged into our devel branch (and within 6 months a new irfu-matlab stable version should be released) to allow for people to get updated.

While the old transition page (https://www.cosmos.esa.int/web/csa/caiototap) still states that:

The following features, already available in CAIO, will be added shortly to the TAP functionality: Header requests (counts as data)

it appears that this functionality may already be available in TAP according to another page (https://www.cosmos.esa.int/web/csa-guide/data-requests#HeaderRequests) so the remaining "meta_data.m" file should be possible to migrate to the TAP interface fairly soon...

thomas-nilsson-irfu commented 2 years ago

Just adding a comment here following an e-mail reminder from ESA CSA received a few minutes ago.

This issue must be fully solved before the 2022-04-14 since at that date the old CLI system CAIO will be turned off and only the TAP interface will be available (as a CLI).

thomas-nilsson-irfu commented 2 years ago

This issue is now considered closed, with the MR84 just having been merged into our devel branch (after a number of successful tests by JanKarlsson). The next few weeks will likely see a new released version of irfu-matlab, that is to say before the final sunset of the old CAIO CLI.