search

SciQLop / speasy

Space Physics made EASY! A simple Python package to deal with main Space Physics WebServices (CDA,SSC,AMDA,..)
Other
24 stars 7 forks source link

Fixes AMDA CSV parser where derived parameters attributes gets overwritten by base param #87

Closed jeandet closed 1 year ago

jeandet commented 1 year ago

Since CSV is still the default format for ADMA with speasy, I felt like we should fix this issue.

brenard-irap commented 1 year ago

Unfortunately, this will not work for some parameters.

The real unit for mms1_fast_jgse is A/m**2, but in the CSV file you have:

# --------------------
# DERIVED PARAMETERS :
# --------------------
#
# PARAMETER_ID : sampling_under_refparam_6474261044550000668
# PARAMETER_NAME : mms1_dis_vgse
# PARAMETER_SHORT_NAME : v_gse
# PARAMETER_UNITS : km/s
# PARAMETER_PROCESS_INFO : Resampling of 'mms1_fpi_dismoms_mms1_dis_bulkv_gse_fast' under times list of 'mms1_fpi_desmoms_mms1_des_numberdensity_fast'
# PARAMETER_LINKED_PARAMS : mms1_fpi_dismoms_mms1_dis_bulkv_gse_fast
#
#
# PARAMETER_ID : sampling_under_refparam_4907928269817716892
# PARAMETER_NAME : mms1_dis_ni
# PARAMETER_SHORT_NAME : density
# PARAMETER_UNITS : cm^-3
# PARAMETER_PROCESS_INFO : Resampling of 'mms1_fpi_dismoms_mms1_dis_numberdensity_fast' under times list of 'mms1_fpi_desmoms_mms1_des_numberdensity_fast'
# PARAMETER_LINKED_PARAMS : mms1_fpi_dismoms_mms1_dis_numberdensity_fast
#
#
# PARAMETER_ID : sampling_under_refparam_3838272296942193163
# PARAMETER_NAME : mms1_des_ne
# PARAMETER_SHORT_NAME : density
# PARAMETER_UNITS : cm^-3
# PARAMETER_PROCESS_INFO : Resampling of 'mms1_fpi_desmoms_mms1_des_numberdensity_fast' under times list of 'mms1_fpi_desmoms_mms1_des_numberdensity_fast'
# PARAMETER_LINKED_PARAMS : mms1_fpi_desmoms_mms1_des_numberdensity_fast
#
#
# PARAMETER_ID : sampling_under_refparam_12014654095256152599
# PARAMETER_NAME : mms1_des_vgse
# PARAMETER_SHORT_NAME : v_gse
# PARAMETER_UNITS : km/s
# PARAMETER_PROCESS_INFO : Resampling of 'mms1_fpi_desmoms_mms1_des_bulkv_gse_fast' under times list of 'mms1_fpi_desmoms_mms1_des_numberdensity_fast'
# PARAMETER_LINKED_PARAMS : mms1_fpi_desmoms_mms1_des_bulkv_gse_fast
#
#
# PARAMETER_ID : mms1_fast_jgse
# PARAMETER_NAME : mms1_fast_jgse
# PARAMETER_SHORT_NAME : j_gse
# PARAMETER_UNITS : A/m**2
# PARAMETER_PROCESS_INFO : Derived parameter from expression '1.6e-10*(#sampling_under_refparam($mms1_des_ne;mms1_des_ne)+#sampling_under_refparam($mms1_dis_ni;mms1_des_ne))*0.5*(#sampling_under_refparam($mms1_dis_vgse;mms1_des_ne)-#sampling_under_refparam($mms1_des_vgse;mms1_des_ne))'
# PARAMETER_PROCESS_DESC : 1.6e-10*(#sampling_under_refparam($mms1_des_ne;mms1_des_ne)+#sampling_under_refparam($mms1_dis_ni;mms1_des_ne))*0.5*(#sampling_under_refparam($mms1_dis_vgse;mms1_des_ne)-#sampling_under_refparam($mms1_des_vgse;mms1_des_ne))
# PARAMETER_LINKED_PARAMS : sampling_under_refparam_12014654095256152599,sampling_under_refparam_3838272296942193163,sampling_under_refparam_4907928269817716892,sampling_under_refparam_6474261044550000668
#
#
# -----------------
# BASE PARAMETERS :
# -----------------
#
# MISSION_ID : NONE
#
#   INSTRUMENT_ID : NONE
#
#     DATASET_ID : mms1-fpi-dismoms
#     DATASET_NAME : mms1:fpi:dismoms
#     DATASET_SOURCE : CDPP/DDServer
#     DATASET_GLOBAL_START : 2015-07-15T15:11:27.381
#     DATASET_GLOBAL_STOP : 2023-04-01T10:45:11.388
#     DATASET_MIN_SAMPLING : 4
#     DATASET_MAX_SAMPLING : 4
#
#       PARAMETER_ID : mms1_dis_ni
#       PARAMETER_NAME : mms1_dis_ni
#       PARAMETER_SHORT_NAME : density
#       PARAMETER_UNITS : cm^-3
#       PARAMETER_TENSOR_ORDER : 0
#       PARAMETER_FILL_VALUE : nan
#
#       PARAMETER_ID : mms1_dis_vgse
#       PARAMETER_NAME : mms1_dis_vgse
#       PARAMETER_SHORT_NAME : v_gse
#       PARAMETER_COMPONENTS : vx,vy,vz
#       PARAMETER_UNITS : km/s
#       PARAMETER_COORDINATE_SYSTEM : GSE
#       PARAMETER_TENSOR_ORDER : 0
#       PARAMETER_FILL_VALUE : nan
#
#     DATASET_ID : mms1-fpi-desmoms
#     DATASET_NAME : mms1:fpi:desmoms
#     DATASET_SOURCE : CDPP/DDServer
#     DATASET_GLOBAL_START : 2015-07-15T15:11:27.381
#     DATASET_GLOBAL_STOP : 2023-04-01T10:45:11.388
#     DATASET_MIN_SAMPLING : 4
#     DATASET_MAX_SAMPLING : 4
#
#       PARAMETER_ID : mms1_des_ne
#       PARAMETER_NAME : mms1_des_ne
#       PARAMETER_SHORT_NAME : density
#       PARAMETER_UNITS : cm^-3
#       PARAMETER_TENSOR_ORDER : 0
#       PARAMETER_FILL_VALUE : nan
#
#       PARAMETER_ID : mms1_des_vgse
#       PARAMETER_NAME : mms1_des_vgse
#       PARAMETER_SHORT_NAME : v_gse
#       PARAMETER_COMPONENTS : vx,vy,vz
#       PARAMETER_UNITS : km/s
#       PARAMETER_COORDINATE_SYSTEM : GSE
#       PARAMETER_TENSOR_ORDER : 0
#       PARAMETER_FILL_VALUE : nan
#

But with this PR, the unit will be km/s in Speasy...

The main problem is that AMDA use an unordered list (I think) to store the list of processes applied to compute the derived parameter. So the unit to use is not necessarily the first or the last available in the header of the CSV file. I will try to fix this in AMDA side.

jeandet commented 1 year ago

@brenard-irap this seems to indicate that the right approach is to extract blocks and find the one with the right PARAMETER_ID.