When requesting data from MARS, if the amount requested does not match the amount available it will fail. For our backfills we want to pull the data even if it is missing a few parameters, so we need to only request what is there. In order to not have to do this manually, first a LIST call is made to MARS to see what is present from the request, and this file is crudely parsed to check for the presence of the desired parameter strings.
This list call returns file sizes as well as parameter names, and as such we have found an edge case where a file size exactly matches a parameter string, so the consumer thinks it is present even though it is missing, makes the RETRIEVE query, and fails.
As such, implement a better parsing system for the file returned from LIST to properly pick out available parameters.
When requesting data from MARS, if the amount requested does not match the amount available it will fail. For our backfills we want to pull the data even if it is missing a few parameters, so we need to only request what is there. In order to not have to do this manually, first a LIST call is made to MARS to see what is present from the request, and this file is crudely parsed to check for the presence of the desired parameter strings.
This list call returns file sizes as well as parameter names, and as such we have found an edge case where a file size exactly matches a parameter string, so the consumer thinks it is present even though it is missing, makes the RETRIEVE query, and fails.
As such, implement a better parsing system for the file returned from LIST to properly pick out available parameters.