search

mmindje / Data-management-FRC-RBIS

The projects under this repository are only for biodiversity data management.
0 stars 0 forks source link

Amphibian occurrence data and MasterList re-tested #43

Closed Thacien1 closed 3 years ago

Thacien1 commented 3 years ago

Dear @helendallas ,

The Team was re-testing the amphibian module on the RBIS testing site. The data has been uploaded successfully. Kindly have a look at the following images.

1) Amphibian module before retesting ( Amphibian dashboard before re-testing

2) Amphibian module after retesting ( Amphibian module re-test 8 April 2021

3) After GBIF harvest

(Amphibian module re-test_GBIF

Below are recleaned and final data sets we have used during module testing.

RBIS Amphibian Master List 2021_4_08 final for RBIS V1_Updated.xlsx RBIS.Amphibian.Data.2020_10_09.Draft.for.RBIS.V1_Updated on 8th APRIL 2021 (1).xlsx

We are waiting for your comment on the next step.

Warm regards,

BDMT

helendallas commented 3 years ago

@Thacien1

Please confirm what was changed in the Master list?

And data file?

mmindje commented 3 years ago

Hi @helendallas

We changed the Origin of amphibian to Native for All. and Three species endemism to Regional endemic level 2 (endemic to one primary catchment. Hyperolius jackie__, H. rwandae and _Boulengerula fischeri. Also, some species were said to not be recorded in GBIF yet they are are all in GBIF.

in the occurence data, we only retested the original file without changing anything.,

helendallas commented 3 years ago

@mmindje Thanks. To confirm - did you delete any records in the master list?

Not sure I understand: Also, some species were said to not be recorded in GBIF yet they are are all in GBIF.

mmindje commented 3 years ago

Thank you Helen. We deleted no records rather rectified the existing ones.

On Tue, 20 Apr 2021 at 10:36, Helen Dallas @.***> wrote:

@mmindje https://github.com/mmindje Thanks. To confirm - did you delete any records in the master list?

Not sure I understand: Also, some species were said to not be recorded in GBIF yet they are are all in GBIF.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mmindje/Data-management-FRC-RBIS/issues/43#issuecomment-823091964, or unsubscribe https://github.com/notifications/unsubscribe-auth/APPTV6VZO4ZMZTCCFLDD4UDTJU4HLANCNFSM42TGVEYQ .

--

Mapendo MINDJE University of Rwanda-Huye Campus

helendallas commented 3 years ago

@mmindje Thanks

helendallas commented 3 years ago

When adding endemism to the Master List always use the shortened version: So Widespread not Widespread (more than one Freshwater Ecoregion) Regional endemic level 2 not Regional endemic level 2 (endemic to one primary catchment)

If the long version is used then it does not look nice in the Dashboards image

I have corrected these and in order to figure out why numbers are different I have removed Test Amphibs from testing and will reingest. Do you know if we ever harvested amphibian data from GBIF? This may be why it is different. Let me check after reingestion on the testing site and get back to you.

helendallas commented 3 years ago

@mmindje Some observation on the revised data file:

I have copied Site description to Original Wetland Name, so that this field is not blank. Can you team, go through these and edit so that for example:

Bugesera district-Mugesera wetland becomes Mugesera Kigali city, Nyandungu wetland becomes Nyandungu etc. so that we have names for wetlands where possible.

Please make a note to do this for all your sites going forward when it is possible to use the Site description to generate a wetland name.

Date needed to be formatted correctly: yyyy/mm/dd - I have changed this

Sampling method: all lowercase: - I have changed this Visual encounter survey

Revised data file for your team to fix Original Wetland Name column is attached: RBIS.Amphibian.Data.2020_10_09.Final.for.RBIS.V1_Updated.on.20.APRIL.2021.1.xlsx

helendallas commented 3 years ago

@mmindje I was checking through the data file uploaded to RBIS live in November, and it only has 352 records, which matches with what is on RBIS dashboard.

This file is attached.

RBIS Amphibian Data 2020_11_25 Final for RBIS V1.xlsx

The new amphibian occurrence file has 1366 records. So data has changed....Please can you check and revert. More data is not a problem, we just need to be sure that the UUID of the original 352 records is the same so that duplicate records are not generated.

helendallas commented 3 years ago

@mmindje I had a quick check of UUIDs and it seems like they may have changed. If this is the case we will need to ask Dimas to delete the amphibian data so that we can re-upload the entire data file with all 1366 records. Can your team please double check the UUIDs.

mmindje commented 3 years ago

Dear @helendallas

We have revised again all the comments given above and we realized that the file we were using for retesting ws the former one with wetlands outside of Rwandan Boundaries. 7 wetlands were outside.

Now, this one attached is the final one with the site description/Original wetland edited as suggested. We did not change the UUID and we think now, we need to re-ingest it in the RBIS after Dimas has removed the amphibian data.

We thank you for your assistance to help us getting now the amphibian file ready. We attach the master list and occurrence data files to be re-ingested in the final site.

RBIS Amphibian Master List 2021_4_08 final for RBIS V1_Revised and Updated on 20th April 2021.xlsx

RBIS.Amphibian.Data.2020_11_25.Final.for.RBIS.V1 Revised and updated on 20th April 2021.xlsx

mmindje commented 3 years ago

When adding endemism to the Master List always use the shortened version: So Widespread not Widespread (more than one Freshwater Ecoregion) Regional endemic level 2 not Regional endemic level 2 (endemic to one primary catchment)

If the long version is used then it does not look nice in the Dashboards image

I have corrected these and in order to figure out why numbers are different I have removed Test Amphibs from testing and will reingest. Do you know if we ever harvested amphibian data from GBIF? This may be why it is different. Let me check after reingestion on the testing site and get back to you.

Thank you @helendallas, On the first trial, we did not harvest the GBIF data. After adding the 352 records, we can also harvest the one from GBIF

mmindje commented 3 years ago

Dear Helen, with the new retesting using updated files, we harvested the GBIF data as results are like this from the 352 to 622 records image

helendallas commented 3 years ago

@mmindje @Thacien1 Please confirm if any amphibian data was actually changed. Dimas is very busy so if the data file is the same, then lets not ask him to remove it.

mmindje commented 3 years ago

Thank you Helen. The changed items included amphibian origin and endemism. In addition, some spp in the materlist were said to not be in GBIF yet they are there. In the occurrence data file, the Site description was also rectified and made as the Original wetland name. We need to update both files.

My advice would be, to reingest both master list and occurrence data files in the RBIS again and harvest the GBIF data.

Attached are the final files to use.

Thank you @Helen

On Wed, 21 Apr 2021 at 10:45, Helen Dallas @.***> wrote:

@mmindje https://github.com/mmindje @Thacien1 https://github.com/Thacien1 Please confirm if any amphibian data was actually changed. Dimas is very busy so if the data file is the same, then lets not ask him to remove it.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mmindje/Data-management-FRC-RBIS/issues/43#issuecomment-823892613, or unsubscribe https://github.com/notifications/unsubscribe-auth/APPTV6X22RVBPFVPRSTIRYLTJ2GEBANCNFSM42TGVEYQ .

--

Mapendo MINDJE University of Rwanda-Huye Campus

helendallas commented 3 years ago

The master file has already been updated in the RBIS live. The origin and endemism of the taxa merely get overwritten, so no need to delete and reingest.

image

For the data file changes I meant to add the wetland name, but retain the Site description as this is what is shown. I do not think it necessary to reingest.

image

Please go through the data on RBIS live and confirm that you are happy. It should all be correct.

mmindje commented 3 years ago

Thank you Helen, I think we can also harvest the data from GBIF Thank you

On Wed, 21 Apr 2021 at 11:49, Helen Dallas @.***> wrote:

The master file has already been updated in the RBIS live. The origin and endemism of the taxa merely get overwritten, so no need to delete and reingest.

[image: image] https://user-images.githubusercontent.com/45286876/115533989-99db9c80-a297-11eb-9cfe-607856dd74f3.png

For the data file changes I meant to add the wetland name, but retain the Site description as this is what is shown. I do not think it necessary to reingest.

[image: image] https://user-images.githubusercontent.com/45286876/115533838-6d278500-a297-11eb-820c-6c9ceabbfb7c.png

Please go through the data on RBIS live and confirm that you are happy. It should all be correct.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mmindje/Data-management-FRC-RBIS/issues/43#issuecomment-823933033, or unsubscribe https://github.com/notifications/unsubscribe-auth/APPTV6U4JOH4ZH56KCRTL7TTJ2NUBANCNFSM42TGVEYQ .

--

Mapendo MINDJE University of Rwanda-Huye Campus

helendallas commented 3 years ago

@mi Agreed. I have set it going, should be done soon as only 54 species.

mmindje commented 3 years ago

@helendallas Thank you for the updated version of amphibs. I think we can also harvest the GBIF data

mmindje commented 3 years ago

Yes! There are only 54 species.

On Wed, 21 Apr 2021 at 13:20, Helen Dallas @.***> wrote:

@mi https://github.com/mi Agreed. I have set it going, should be done soon as only 54 species.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mmindje/Data-management-FRC-RBIS/issues/43#issuecomment-823983534, or unsubscribe https://github.com/notifications/unsubscribe-auth/APPTV6V7WEHSRJZNFAPPZK3TJ2YGVANCNFSM42TGVEYQ .

--

Mapendo MINDJE University of Rwanda-Huye Campus