In November 2023, we found that the field scientificName was appearing twice in some downloads. This turned out to happen because verbatimScientificName had been attached to the Darwin Core term scientificName on the atlas side, resulting in duplication.
Our solution - which we intended to be temporary - was to store user-supplied names in select.data_request() and galah_select(), then use these to overwrite server-supplied names. These changes were applied in this commit and shipped with version 2.0.0.
The risk with this approach is that any change to the number or sequence of fields in the download will break field names, and possibly fail to return anything at all. It now appears that users with authority to see 'sensitive' data (i.e. via the SDS system) receive additional fields, and potentially in a different order, from those with 'general' access. This needs fixing, but also including in tests in a 'safe' way (i.e. without allowing general access to sensitive records).
mjwestgate
commented
2 months ago
In November 2023, we found that the field
scientificNamewas appearing twice in some downloads. This turned out to happen becauseverbatimScientificNamehad been attached to the Darwin Core termscientificNameon the atlas side, resulting in duplication.Our solution - which we intended to be temporary - was to store user-supplied names in
select.data_request()andgalah_select(), then use these to overwrite server-supplied names. These changes were applied in this commit and shipped with version 2.0.0.The risk with this approach is that any change to the number or sequence of fields in the download will break field names, and possibly fail to return anything at all. It now appears that users with authority to see 'sensitive' data (i.e. via the SDS system) receive additional fields, and potentially in a different order, from those with 'general' access. This needs fixing, but also including in tests in a 'safe' way (i.e. without allowing general access to sensitive records).