search

gbif / ingestion-management

Tracking of data issues seen during data ingestion processes
Apache License 2.0
1 stars 0 forks source link

Identifiers validation failed for dataset Texas A&M University Insect Collection #7

Closed gbif-pipelines closed 1 year ago

gbif-pipelines commented 1 year ago

Identifier validation failed for the dataset Texas A&M University Insect Collection:



- Pipelines execution steps: [http://api.gbif.org/v1/pipelines/history/execution/3029849/step](http://api.gbif.org/v1/pipelines/history/execution/3029849/step)
<details>
<summary>Publisher email</summary>

Hello,

I am contacting you from the GBIF Secretariat about a dataset published by the [Texas A&M University Insect Collection](https://registry.gbif.org/dataset/96193ea2-f762-11e1-a439-00145eb45e9a) : https://doi.org/10.15468/caprqh.
We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.

As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence.
This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.

In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.

We would like to check with you if those changes were intentional. Do you know if this is the case?
Please let us know, thanks! We are happy to resume the dataset ingestion.

Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve
the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item).
If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?

Please let us know if you have any question. Thanks!

All the best,

</details>

You can skip/fix identifier validation using [the registry UI](https://registry.gbif.org/dataset/96193ea2-f762-11e1-a439-00145eb45e9a/ingestion-history).
ManonGros commented 1 year ago

Cecilie contacted the Symbiotic SCAN portal manager about this dataset on March, 29th. The answer at the time was that the file was corrupt but they weren't available to fix things at the moment. They wrote to bring it back up later if needed.

gbif-pipelines commented 1 year ago

Identifier validation still failing for the dataset Texas A&M University Insect Collection:

Hello,

I am contacting you from the GBIF Secretariat about a dataset published by the Texas A&M University Insect Collection : https://doi.org/10.15468/caprqh. We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.

As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence. This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.

In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.

We would like to check with you if those changes were intentional. Do you know if this is the case? Please let us know, thanks! We are happy to resume the dataset ingestion.

Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item). If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?

Please let us know if you have any question. Thanks!

All the best,

You can skip/fix identifier validation using the registry UI.

gbif-pipelines commented 1 year ago

Identifier validation still failing for the dataset Texas A&M University Insect Collection:

Hello,

I am contacting you from the GBIF Secretariat about a dataset published by the Texas A&M University Insect Collection : https://doi.org/10.15468/caprqh. We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.

As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence. This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.

In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.

We would like to check with you if those changes were intentional. Do you know if this is the case? Please let us know, thanks! We are happy to resume the dataset ingestion.

Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item). If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?

Please let us know if you have any question. Thanks!

All the best,

You can skip/fix identifier validation using the registry UI.

gbif-pipelines commented 1 year ago

Identifier validation still failing for the dataset Texas A&M University Insect Collection:

Hello,

I am contacting you from the GBIF Secretariat about a dataset published by the Texas A&M University Insect Collection : https://doi.org/10.15468/caprqh. We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.

As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence. This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.

In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.

We would like to check with you if those changes were intentional. Do you know if this is the case? Please let us know, thanks! We are happy to resume the dataset ingestion.

Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item). If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?

Please let us know if you have any question. Thanks!

All the best,

You can skip/fix identifier validation using the registry UI.

gbif-pipelines commented 1 year ago

Identifier validation still failing for the dataset Texas A&M University Insect Collection:

Hello,

I am contacting you from the GBIF Secretariat about a dataset published by the Texas A&M University Insect Collection : https://doi.org/10.15468/caprqh. We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.

As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence. This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.

In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.

We would like to check with you if those changes were intentional. Do you know if this is the case? Please let us know, thanks! We are happy to resume the dataset ingestion.

Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item). If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?

Please let us know if you have any question. Thanks!

All the best,

You can skip/fix identifier validation using the registry UI.

gbif-pipelines commented 1 year ago

Identifier validation still failing for the dataset Texas A&M University Insect Collection:

Hello,

I am contacting you from the GBIF Secretariat about a dataset published by the Texas A&M University Insect Collection : https://doi.org/10.15468/caprqh. We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset.

As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence. This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated.

In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets.

We would like to check with you if those changes were intentional. Do you know if this is the case? Please let us know, thanks! We are happy to resume the dataset ingestion.

Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item). If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option?

Please let us know if you have any question. Thanks!

All the best,

You can skip/fix identifier validation using the registry UI.

gbif-pipelines commented 1 year ago

Identifier validation still failing for the dataset Texas A&M University Insect Collection:

New IDs sample:

21419238
15247807
15247682
15240327
15247797
15237253
15247801
15247808
15247792
21420106
Old IDs sample:

7879b90b-58ed-4e0a-b6c9-498f4f28b7fb
94ab3c4c-108c-4df7-99e8-99cda2b6d6cd
3d44fecb-b514-4b9e-9738-a64b0e6251de
f93bb67f-39a9-40e0-a7a7-da7f9691b702
c70a99c5-650d-49e8-8403-4724f1565495
34f8c9c7-6edc-42aa-aded-fc8846c14322
5c0dcad1-d57e-4186-a235-d4721d8ebbeb
f7367624-e12a-4d56-9b56-30b3a710355a
5410651e-7d2b-4624-b6a2-37a4b270631b
04ff8ff2-2c67-4504-8442-296793b4b92d
Publisher email Hello, I am contacting you from the GBIF Secretariat about a dataset published by the [Texas A&M University Insect Collection](https://registry.gbif.org/dataset/96193ea2-f762-11e1-a439-00145eb45e9a) : https://doi.org/10.15468/caprqh. We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset. As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence. This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated. In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets. We would like to check with you if those changes were intentional. Do you know if this is the case? Please let us know, thanks! We are happy to resume the dataset ingestion. Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item). If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option? Please let us know if you have any question. Thanks! All the best,

You can skip/fix identifier validation using the registry UI.

gbif-pipelines commented 1 year ago

Identifier validation still failing for the dataset Texas A&M University Insect Collection:

New IDs sample:

21422310
15247810
15240422
15247672
15237254
15247790
15247789
15237257
15247674
15247798
Old IDs sample:

7879b90b-58ed-4e0a-b6c9-498f4f28b7fb
94ab3c4c-108c-4df7-99e8-99cda2b6d6cd
3d44fecb-b514-4b9e-9738-a64b0e6251de
f93bb67f-39a9-40e0-a7a7-da7f9691b702
c70a99c5-650d-49e8-8403-4724f1565495
34f8c9c7-6edc-42aa-aded-fc8846c14322
5c0dcad1-d57e-4186-a235-d4721d8ebbeb
f7367624-e12a-4d56-9b56-30b3a710355a
5410651e-7d2b-4624-b6a2-37a4b270631b
04ff8ff2-2c67-4504-8442-296793b4b92d
Publisher email Hello, I am contacting you from the GBIF Secretariat about a dataset published by the [Texas A&M University Insect Collection](https://registry.gbif.org/dataset/96193ea2-f762-11e1-a439-00145eb45e9a) : https://doi.org/10.15468/caprqh. We noticed that the occurrenceIDs were changed. We have temporarily paused the ingestions of this dataset. As you might already know, when an occurrence record has a new occurrenceID for a given dataset, our system considers it to be a new occurrence. This means that it will be given a new gbifid and a new occurrence URL (like this one: https://www.gbif.org/occurrence/1252968762) and the old gbifid and URL will be deprecated. In this case, this means that the occurrence URLs would be deprecated when ingesting the newest versions of these datasets. We would like to check with you if those changes were intentional. Do you know if this is the case? Please let us know, thanks! We are happy to resume the dataset ingestion. Note that some users rely on those occurrence URLs and gbifids (like https://bionomia.net for example). In an attempt to improve the stability of the occurrence URLs and gbifids, we have implemented a warning system to detect these type of changes in datasets (see this news item). If the data publisher can provide us with a list of old and new occurrenceIDs per record, we can avoid the identifier and URL changes. Could that be an option? Please let us know if you have any question. Thanks! All the best,

You can skip/fix identifier validation using the registry UI.

ManonGros commented 1 year ago

It has been more than 3 months and although the publisher mentioned trying to fix the issue, the dataset is still paused. I am letting it go through.