search

kartoza / django-bims

https://testing.healthyrivers.kartoza.com
GNU Affero General Public License v3.0
12 stars 19 forks source link

Non Validated data #2439

Closed helendallas closed 3 years ago

helendallas commented 3 years ago

There are 122 sites that show Non Validated data. Some of these have been deleted as they were test data, but they are still visible when the filter is applied. I am checking these now, but please can you also see what is happening

image

dimasciput commented 3 years ago

Hi Helen,

When you delete data from the system, the search results are not updated automatically, because they are temporarily stored. You basically need to wait 10 minutes to see the updated search results.

On Mon, 7 Dec 2020 at 18:58, Helen Dallas notifications@github.com wrote:

Assigned #2439 https://github.com/kartoza/django-bims/issues/2439 to @dimasciput https://github.com/dimasciput.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/kartoza/django-bims/issues/2439#event-4079254969, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPDJMIF3562RMD22H3QC2TSTS7NZANCNFSM4UQLKTSQ .

-- Dimas Ciputra - Software Developer Email : dimas@kartoza.com Tel : +62 812 1679 2585 Visit https://kartoza.com to find out about open source : • Desktop GIS programming services • Geospatial web development • GIS Training • Consulting Services

helendallas commented 3 years ago

@dimasciput I checked and there are still 122 sites with data that are non-validated.
image

However when I go to Validate Records - there is nothing to validate.
image

Please can we discuss validation workflows as I am not sure these are working correctly.

We need to confirm what data needs to be validated and have these shown on the Validate Records.

dimasciput commented 3 years ago

@helendallas the non validated data now appear in https://freshwaterbiodiversity.org/nonvalidated-list/

helendallas commented 3 years ago

@dimasciput The validation workflow is not practical as it is. I have 180 pages of individual taxa to validate. As we spoke about previously and see #2460 We need to validate data per Site Visit, not per taxon, especially for SASS data added. We also need to be able to validate a Site.

Please give some thought to this so we can action sooner than later as more and more data are being added. @timlinux @jeremyshelton

dimasciput commented 3 years ago

@helendallas this is fixed and deployed to live.

helendallas commented 3 years ago

@dimasciput The data shown here is strange. Its my data, and came in with the Rivers databse, but I am not seeing any of the new data that users have added to FBIS. I should see all non-validated data

image

helendallas commented 3 years ago

also the sampling date says 4 dec 2020, but its not image

helendallas commented 3 years ago

@dimasciput What is happening is that it is filtering by my name when I hit See unvalidtaed data -this should not happen as the only way to validate is to see all unvalidated data regardless of the owner.

helendallas commented 3 years ago

So most dates do not match between SASS record and that listed on the site visit for in the column Sampling date

helendallas commented 3 years ago

@dimasciput Please can we discuss this tomorrow.

When I click Validate records on Administration, I need to see all Unvalidated data. Then I need to be able to apply a filter to select a subset. What is missing but which is critical is Module - as the person checking fish data will be different to invertebrate data.

Also I am not sure why there is the Send for Validation button? For users this should be automatics, and for validators, they dont need to see this as that is the purpose of the Validate record admin function.

Please cna you also check why quite a lot of Rivers data is now showing as unvalidated

dimasciput commented 3 years ago

When I click Validate records on Administration, I need to see all Unvalidated data.

Will remove the owner filter

What is missing but which is critical is Module - as the person checking fish data will be different to invertebrate data.

Will add this

Also I am not sure why there is the Send for Validation button?

Will remove this process

helendallas commented 3 years ago

@dimasciput PLease keep filter Owner, but when it is blank, then all unvalidated records must be shown, The option to filter on owner is useful but should be optional

helendallas commented 3 years ago

@dimasciput I have also noticed that the sampling dates are wrong...

image

dimasciput commented 3 years ago

@dimasciput I have also noticed that the sampling dates are wrong...

@helendallas which one is not correct?

helendallas commented 3 years ago

The dates were different to what was on the sheets. Since I have already validated these they are gone now.

dimasciput commented 3 years ago

@helendallas I have updated this, please check it again.

helendallas commented 3 years ago

@dimasciput I am not seeing any unvalidated data now?

https://freshwaterbiodiversity.org/site-visit/list/?validated=[%22non%20validated%22]&o=date

dimasciput commented 3 years ago

@helendallas I think I misunderstand something, so I validated existing data for now. But the new data will come up in the unvalidated, is that okay?

dimasciput commented 3 years ago

@helendallas Or should I set all user added data to be unvalidated?

helendallas commented 3 years ago

@dimasciput

We still need to validate data that users have uploaded directly to FBIS (SASS, fish) ......is it possible to show these?

All data uploaded by admin, or imported from rivers or biobase should be validated.

helendallas commented 3 years ago

Yes, please set all user added data to unvalidated. I assume data uploaded by admin using the upload csv will still be validated?

dimasciput commented 3 years ago

@helendallas yep updated, please check

helendallas commented 3 years ago

Thanks @dimasciput Looking good, a few issues I have noticed: Owner still not coming through, sometimes blank even though there is an owner - these are all me e.g. image

https://freshwaterbiodiversity.org/sass/view/2376/

However when I apply the owner filter as well then my name is there

https://freshwaterbiodiversity.org/site-visit/list/?validated=[%22non%20validated%22]&o=date&modules=21&collectors=%5B478%5D

2) Capturer is still showing - not necessary: e.g. Aneri captured the data but James is the Owner, only need James shown image

other examples: image

helendallas commented 3 years ago

@dimasciput So I recently went through the validation process, Dale Kindler had added both SASS data (correct) and invert data that was the same, so duplicate site visits. I deleted the non-SASS data, and then went back to validate the correct SASS data, but no longer have the option.

image

https://freshwaterbiodiversity.org/site-visit/list/?validated=[%22non%20validated%22]&o=date&site_code=A2APIE-ERASM

dimasciput commented 3 years ago

@helendallas how did you delete the non-sass data?

dimasciput commented 3 years ago

you can only delete SASS site visit for now, so probably you deleted the SASS data instead.

helendallas commented 3 years ago

@dimasciput I just rejected it on the validation page, I initially saw sets of data, but sadly didnt take a screenshot,

helendallas commented 3 years ago

Nope the SASS data is still there

https://freshwaterbiodiversity.org/sass/dashboard/31388/?taxon=&search=&siteId=31388&collector=&category=&yearFrom=&yearTo=&months=&boundary=&userBoundary=&referenceCategory=&spatialFilter=&reference=&endemic=&conservationStatus=[]&modules=&validated=[%22non%20validated%22]&sourceCollection=[%22fbis%22]&abioticData=&ecologicalCategory=&rank=&siteIdOpen=31388&orderBy=name&polygon=

dimasciput commented 3 years ago

@helendallas yep I can see the data in the admin page too, will fix it

dimasciput commented 3 years ago

@helendallas apparently there are 5 records that are not SASS data, but captured by Dale. Should I remove them?

helendallas commented 3 years ago

@dimasciput Yes, what happened was he didnt realise SASS data should be captured using the SASS form, and so captured some data using the general invertebrate dashboard. I pointed out his error, but he then couldnt delete these data before capturing the data correctly using the SASS data capture form. Make sense?

dimasciput commented 3 years ago

Yep, let me remove them :+1:

helendallas commented 3 years ago

@dimasciput Can you please include Owner in the sort

image

dimasciput commented 3 years ago

@helendallas might be tricky for that, probably need a new ticket

dimasciput commented 3 years ago

https://freshwaterbiodiversity.org/site-visit/list/?validated=%5B%22non%20validated%22%5D&o=date&site_code=A2APIE-ERASM

This is fixed

helendallas commented 3 years ago

@dimasciput ok if tricky, I will just use filter.

I still don see option to validate....for this site image

dimasciput commented 3 years ago

@helendallas when you reject the data, the data will come back to the owner, and they need to fix it and click "Ready for validation". Once in review mode, you can then approve or reject it again. But for this case, let me just set to in review manually for now.

helendallas commented 3 years ago

Thanks, I rejected the data from the other site that had the same code and sampling date, not this one. but happy for you to do manually so I can validate it.

helendallas commented 3 years ago

Closing this ticket as outstanding issues have their own ticket now

helendallas commented 3 years ago

@dimasciput please check why once I have validated a site, it still shows on the Validate Records list. Surely once FBIS Admin have validated a Record, then the record should be removed, or is it because the Site has not yet been validated. If the latter then we really need the site validation to be finalised on testing and rolled out to live. It is affecting our work. @Fanevanjanahary Where are we with finalising the Site Validation. Please resspond asap.

image

helendallas commented 3 years ago

Closing ticket, not satisfactorily resolved yet though