Listing more than 48 variables does not work properly

[karlmsmith]

This came up from the all-available-variables listing in SOCAT, which has almost 60 variables. But apparently this also comes up in other contexts when requesting many variables in a command.

The ferret command (55 variables):

LIST/APPEND/FILE=tmp.tsv/width=1024/norow/format=tab expocode, latitude,  longitude, socat_doi, all_region_ids, depth, year, month, day, hour, minute, second, sal, temp, Temperature_equi, delta_temp, Temperature_atm, Pressure_atm, Pressure_equi, woa_sss, pressure_ncep_slp, etopo2, dist_to_land, gvCO2, fCO2_recommended, fCO2_source, WOCE_CO2_water, WOCE_CO2_atm, xCO2_water_equi_temp_wet_ppm, xCO2_water_sst_wet_ppm, xCO2_water_equi_temp_dry_ppm, xCO2_water_sst_dry_ppm, fCO2_water_equi_uatm, fCO2_water_sst_100humidity_uatm, pCO2_water_equi_temp, pCO2_water_sst_100humidity_uatm, fCO2_atm_wet_actual, fCO2_atm_wet_interp, pCO2_atm_wet_actual, pCO2_atm_wet_interp, xCO2_atm_dry_actual, xCO2_atm_dry_interp, delta_xCO2, delta_fCO2, delta_pCO2, xH2O_equi, relative_humidity, specific_humidity, ship_speed, ship_dir, wind_speed_rel, wind_speed_true, wind_dir_true, wind_dir_rel, region_id

gives the result (49-55 duplicated, overwriting what should be in 17-23):

             DATA SET: ..._netcdf.nc
             SOCAT Latest Data Collection
             X: 0.5 to 5536.5
             E: 1
 Column  1: EXPOCODE is expocode
 Column  2: LATITUDE is Latitude (degrees_north)   BAD FLAG : -1.E+34
 Column  3: LONGITUDE is Longitude (degrees_east)   BAD FLAG : -1.E+34
 Column  4: SOCAT_DOI is DOI of SOCAT-enhanced data
 Column  5: ALL_REGION_IDS is Sorted unique region IDs
 Column  6: DEPTH is sample depth (m)   BAD FLAG : -1.E+34
 Column  7: YEAR is year    BAD FLAG : -99
 Column  8: MONTH is month of year    BAD FLAG : -99
 Column  9: DAY is day of month    BAD FLAG : -99
 Column 10: HOUR is hour of day    BAD FLAG : -99
 Column 11: MINUTE is minute of hour    BAD FLAG : -99
 Column 12: SECOND is second of minute    BAD FLAG : -1.E+34
 Column 13: SAL is salinity (PSU)   BAD FLAG : -1.E+34
 Column 14: TEMP is sea surface temperature (degrees C)   BAD FLAG : -1.E+34
 Column 15: TEMPERATURE_EQUI is equilibrator chamber temperature (degrees C)   BAD FLAG : -1.E+34
 Column 16: DELTA_TEMP is Equilibrator Temp - SST (degrees C)   BAD FLAG : -1.E+34
 Column 17: SHIP_SPEED is measured ship speed (knots)   BAD FLAG : -1.E+34
 Column 18: SHIP_DIR is ship direction (degrees)   BAD FLAG : -1.E+34
 Column 19: WIND_SPEED_REL is relative wind speed (m/s)   BAD FLAG : -1.E+34
 Column 20: WIND_SPEED_TRUE is true wind speed (m/s)   BAD FLAG : -1.E+34
 Column 21: WIND_DIR_TRUE is true wind direction (degrees)   BAD FLAG : -1.E+34
 Column 22: WIND_DIR_REL is relative wind direction (degrees)   BAD FLAG : -1.E+34
 Column 23: REGION_ID is SOCAT region ID
 Column 24: GVCO2 is GlobalView xCO2 (umol/mol)   BAD FLAG : -1.E+34
 Column 25: FCO2_RECOMMENDED is fCO2 recommended (uatm)   BAD FLAG : -1.E+34
 Column 26: FCO2_SOURCE is Algorithm number for recommended fCO2    BAD FLAG : -99
 Column 27: WOCE_CO2_WATER is WOCE flag for aqueous CO2
 Column 28: WOCE_CO2_ATM is WOCE flag for atmospheric CO2
 Column 29: XCO2_WATER_EQUI_TEMP_WET_PPM is water xCO2 wet using equi temp (umol/mol)   BAD FLAG : -1.E+34
 Column 30: XCO2_WATER_SST_WET_PPM is water xCO2 wet using sst (umol/mol)   BAD FLAG : -1.E+34
 Column 31: XCO2_WATER_EQUI_TEMP_DRY_PPM is water xCO2 dry using equi temp (umol/mol)   BAD FLAG : -1.E+34
 Column 32: XCO2_WATER_SST_DRY_PPM is water xCO2 dry using sst (umol/mol)   BAD FLAG : -1.E+34
 Column 33: FCO2_WATER_EQUI_UATM is water fCO2 wet using equi temp (uatm)   BAD FLAG : -1.E+34
 Column 34: FCO2_WATER_SST_100HUMIDITY_UATM is water fCO2 wet using sst (uatm)   BAD FLAG : -1.E+34
 Column 35: PCO2_WATER_EQUI_TEMP is water pCO2 wet using equi temp (uatm)   BAD FLAG : -1.E+34
 Column 36: PCO2_WATER_SST_100HUMIDITY_UATM is water pCO2 wet using sst (uatm)   BAD FLAG : -1.E+34
 Column 37: FCO2_ATM_WET_ACTUAL is actual air fCO2 wet (uatm)   BAD FLAG : -1.E+34
 Column 38: FCO2_ATM_WET_INTERP is interpolated air fCO2 wet (uatm)   BAD FLAG : -1.E+34
 Column 39: PCO2_ATM_WET_ACTUAL is actual air pCO2 wet (uatm)   BAD FLAG : -1.E+34
 Column 40: PCO2_ATM_WET_INTERP is interpolated air pCO2 wet (uatm)   BAD FLAG : -1.E+34
 Column 41: XCO2_ATM_DRY_ACTUAL is actual air xCO2 dry (umol/mol)   BAD FLAG : -1.E+34
 Column 42: XCO2_ATM_DRY_INTERP is interpolated air xCO2 dry (umol/mol)   BAD FLAG : -1.E+34
 Column 43: DELTA_XCO2 is water xCO2 minus atmospheric xCO2 (umol/mol)   BAD FLAG : -1.E+34
 Column 44: DELTA_FCO2 is water fCO2 minus atmospheric fCO2 (uatm)   BAD FLAG : -1.E+34
 Column 45: DELTA_PCO2 is water pCO2 minus atmospheric pCO2 (uatm)   BAD FLAG : -1.E+34
 Column 46: XH2O_EQUI is xH2O in equil air sample (mmol/mol)   BAD FLAG : -1.E+34
 Column 47: RELATIVE_HUMIDITY is relative humidity    BAD FLAG : -1.E+34
 Column 48: SPECIFIC_HUMIDITY is specific humidity    BAD FLAG : -1.E+34
 Column 49: SHIP_SPEED is measured ship speed (knots)   BAD FLAG : -1.E+34
 Column 50: SHIP_DIR is ship direction (degrees)   BAD FLAG : -1.E+34
 Column 51: WIND_SPEED_REL is relative wind speed (m/s)   BAD FLAG : -1.E+34
 Column 52: WIND_SPEED_TRUE is true wind speed (m/s)   BAD FLAG : -1.E+34
 Column 53: WIND_DIR_TRUE is true wind direction (degrees)   BAD FLAG : -1.E+34
 Column 54: WIND_DIR_REL is relative wind direction (degrees)   BAD FLAG : -1.E+34
 Column 55: REGION_ID is SOCAT region ID

If the number of variables is 48, then all is fine. If 49 given, then one duplicate/overwrite in column 17; If 50, then two duplicate overwrites.

When this has cropped up in other contexts, one can break up the operation so no more the 48 variables are used at a time. But in for these TSV or CSV files, we cannot use this technique.

[karlmsmith]

Debugging xeq_list with 49 variables, the my_cx array (copied from the is_cx array) appears correct

(4, 5, 6, ..., 50, 51, 52, 0, 0, 0, ...)

but the my_mr array (and the is_mr array it is copied from) appears correct only up through variable 48

(1, 2, 3, ..., 47, 48, 17, 0, 0, 0, ...)

So this then causes the listing of the last rows to show up again starting at 17. So appears the problem is in GET_PROT_CMND_DATA which assigns the is_mr array.

[AnsleyManke]

The place where we (possibly) ran into this before is described in #1757. There's a tar file with some scripts and data that produces this.

[karlmsmith]

For this particular case, interp_stack line 336 calls FIND_MEM_VAR_INCL_C_CACHE The call to INTERP_STACK is passing the namecode of the particular variable, which is a string EX#1, EX#2, ... EX#9, EX#:, EX#;, ..., EX#@, EX#A, ..., EX#Z, EX#[, ..., `EX#, then one with the grave, then ,EX#aat isp_base=49. This last name_code matches the uppercase name codeEX#A` at isp_base=17, so it thinks it has found a match.

So I think the solution is either the name_code matching needs to always be case sensitive, or it needs to avoid lowercase letters. Need to investigate further what can be done, but this explains the behavior seen here and probably in numerous other areas with large lists of variables.

[karlmsmith]

It is FIND_MEM_VAR at find_mem_var_incl_c_cache.F line 68 that is thinking it has found the variable. The mv returned by mv_flink(mv) at line 185 (label 100) returns a match (positive integer). However, setting this to zero, indicating no match, does not change the results (ie, still incorrect). Setting mode stupid (which is suppose to disable finding matches in memory) also does change the results.

[karlmsmith]

I see other places where it is assumed the name code is uppercase or is compared case blind. So the lowercase letters definitely need to be eliminated from use. EXPR_NAME is the offending routine - far too simplistic. Need to check whether more than one character after the EX# will be recognized as different.

[AnsleyManke]

I agree that internally comparisons of names need to be case-insensitive. Case sensitivity is handled in just a couple of cases, on dataset initialization for file variables, and when user-defined variable names are stored, so these can be written out using the original upper- or lower-case spelling. The rest of the code should be able to depend on being case-insensitive.

[karlmsmith]

Revised to only use numbers and use more than one characters after the EX#. (The original method would blow up after about 207 variables.) So now EX#1, EX#2, ..., EX#9, EX#10, EX#11, ...., EX#99999. (The name code is CHARACTER*8.) STOP statements added if var number given is less than zero or more than 99999, which should never happen.

No differences observed in benchmark tests. Will add benchmark test from this SOCAT example.

[karlmsmith]

benchmark test added and test results updated https://github.com/NOAA-PMEL/Ferret/commit/87bd7fd7d008a9b12f375c47160c83d86c04a991

[AnsleyManke]

The case in ticket #1757 is not resolved with this change.

That was a highly convoluted script which defines thousands of variables. It'd be nice if it were solved, but there was a straightforward way in that case to reorganize it. We pulled out a small number of variable definitions into a second script and then called that multiple times from the main script.