search

iodepo / OceanBestPractices

Repository to store the OpenSource version of the code made by E84 for OceanBestPractices.org
https://oceanbestpractices.org
GNU Affero General Public License v3.0
12 stars 6 forks source link

new Search interface Stats display stuck at 1705 #257

Closed paulineobps closed 1 year ago

paulineobps commented 2 years ago

new production Search interface Stats display stuck at 1705 - it should be showing 1708 image

paulineobps commented 2 years ago

20 Sep - stats still not displaying the uptodate number (1718) BUT also new deposits from number 1706 onwards are not findable. Email sent directly to Brandon

paulpilone commented 2 years ago

@paulineobps The cron lambda is running but it hasn't indexed any documents since 9/1. I just ran it again and forced it to index the documents currently published to the feed (which worked). I've reset the timer and will keep an eye on whether or not it stops indexing. I'm not 100% sure why it wasn't finding documents despite the dev function working.

paulineobps commented 2 years ago

Paul a special thank you . I dare not ask why on our website **** it shows 1718 and on the search interface 1715 . So far I have been able to find the latest deposits.. Thank you for the fix and fingers crossed the cron continues to work. I will add some new deposits to test.

paulpilone commented 2 years ago

I think that's a side effect of using the RSS feed to ingest documents. It looks like the feed only has at most ~6 documents listed so if the ingester missed some (it didn't index anything since 9/1) and they were removed from the feed we don't have any other way to discover new documents. I can run the bulk indexer later to make sure we discover which 3 were missed.

paulineobps commented 2 years ago

I have added a new deposit and it is findable and the total increased by one, thanks Paul. Is there anything I can do in the deposit flow to better support this feed and index process?

paulineobps commented 2 years ago

I tell a lie! apologies

I can find all the 'old' deposits that were not showing or counted but I cannot find the new deposit from yesterday A Survey of Underwater Acoustic Data Classification Methods Using Deep Learning for Shoreline Surveillance. and the stats have not moved to 1716 (on AWS)

paulpilone commented 2 years ago

@paulineobps I've made changes to the RSS reader that I hope improves its reliability. I'm going to push that to dev now and if everything looks ok I'll eventually push it to prod.

paulpilone commented 2 years ago

@paulineobps Deployed this change to both develop and production. Both are showing 1718 documents now. I think we can close this issue if you feel like new documents are being indexed correctly.

paulpilone commented 2 years ago

Scratch that: prod shows 1715 which is inline with missing a few. I'll run the bulk indexer and it should catch them all. I believe the changes I made to the RSS reader should fix missing documents as they're published.

paulineobps commented 2 years ago

Thanks Paul really appreciate you getting this sorted. Fingers crossed the bulk indexer gets us to 1719 and we can search and find the last deposit (A Survey of Underwater Acoustic Data Classificationhttps://repository.oceanbestpractices.org/handle/11329/2072 ...)

I have a hurricane soon to loom over us, so am rushing around with 'battening down the hatches'. No reply to your Github messages is because of that only. Arno is working in Barcelona for a week, so I will try to contact him re the permissions and access questions you posed.

Pauline Pauline SIMPSON Ocean Best Practices System Project Manager https://www.oceanbestpractices.org/

UNESCO /IOC Project Office for IODE, InnovOcean Campus – Jacobsenstraat 1 8400 Oostende, Belgium Email: @.**@.>; Alternate email: @.**@.>; Skype: Pauline Simpson ORCID: 0000-0003-2551-5740

http://www.iode.org

From: Paul Pilone @.> Sent: 23 September 2022 14:25 To: iodepo/OceanBestPractices @.> Cc: Simpson, Pauline @.>; Mention @.> Subject: Re: [iodepo/OceanBestPractices] new Search interface Stats display stuck at 1705 (Issue #257)

CAUTION: This email is external from UNESCO. Please be vigilant on its sender and content. ATTENTION : Cet e-mail est externe à l'UNESCO. Soyez vigilant sur son expéditeur et contenu.

Scratch that: prod shows 1715 which is inline with missing a few. I'll run the bulk indexer and it should catch them all. I believe the changes I made to the RSS reader should fix missing documents as they're published.

— Reply to this email directly, view it on GitHubhttps://github.com/iodepo/OceanBestPractices/issues/257#issuecomment-1256593942, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFZQUQZLD2BIA7M5AH3NBLDV7X73XANCNFSM6AAAAAAQDHGTDQ. You are receiving this because you were mentioned.Message ID: @.***>

paulineobps commented 2 years ago

AWS/EDS still showing 1715 so no regular updating

paulpilone commented 2 years ago

@paulineobps were these new documents that were published to the RSS feed? I see that the feed ingester ran today and queued a few documents up for indexing:

image
paulineobps commented 2 years ago

Thanks Paul . This is the last one input yesterday https://repository.oceanbestpractices.org/handle/11329/2073 at

2022-09-27T21:20:42Z

Pauline Pauline SIMPSON Ocean Best Practices System Project Manager https://www.oceanbestpractices.org/

UNESCO /IOC Project Office for IODE, InnovOcean Campus – Jacobsenstraat 1 8400 Oostende, Belgium Email: @.**@.>; Alternate email: @.**@.>; Skype: Pauline Simpson ORCID: 0000-0003-2551-5740

http://www.iode.org

From: Paul Pilone @.> Sent: 28 September 2022 11:26 To: iodepo/OceanBestPractices @.> Cc: Simpson, Pauline @.>; Mention @.> Subject: Re: [iodepo/OceanBestPractices] new Search interface Stats display stuck at 1705 (Issue #257)

CAUTION: This email is external from UNESCO. Please be vigilant on its sender and content. ATTENTION : Cet e-mail est externe à l'UNESCO. Soyez vigilant sur son expéditeur et contenu.

@paulineobpshttps://github.com/paulineobps were these new documents that were published to the RSS feed? I see that the feed ingester ran today and queued a few documents up for indexing:

[image]https://user-images.githubusercontent.com/981227/192834360-484cfb0d-c5b4-4d2e-9a21-377e4e04d574.png

— Reply to this email directly, view it on GitHubhttps://github.com/iodepo/OceanBestPractices/issues/257#issuecomment-1261155746, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFZQUQ3QLNUWQPHF7GQ3V23WARWTZANCNFSM6AAAAAAQDHGTDQ. You are receiving this because you were mentioned.Message ID: @.***>

paulineobps commented 2 years ago

Still showing 1715 and search for latest deposits does not find the titles

Pauline Pauline SIMPSON Ocean Best Practices System Project Manager https://www.oceanbestpractices.org/

UNESCO /IOC Project Office for IODE, InnovOcean Campus – Jacobsenstraat 1 8400 Oostende, Belgium Email: @.**@.>; Alternate email: @.**@.>; Skype: Pauline Simpson ORCID: 0000-0003-2551-5740

http://www.iode.org

From: Paul Pilone @.> Sent: 23 September 2022 14:25 To: iodepo/OceanBestPractices @.> Cc: Simpson, Pauline @.>; Mention @.> Subject: Re: [iodepo/OceanBestPractices] new Search interface Stats display stuck at 1705 (Issue #257)

CAUTION: This email is external from UNESCO. Please be vigilant on its sender and content. ATTENTION : Cet e-mail est externe à l'UNESCO. Soyez vigilant sur son expéditeur et contenu.

Scratch that: prod shows 1715 which is inline with missing a few. I'll run the bulk indexer and it should catch them all. I believe the changes I made to the RSS reader should fix missing documents as they're published.

— Reply to this email directly, view it on GitHubhttps://github.com/iodepo/OceanBestPractices/issues/257#issuecomment-1256593942, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFZQUQZLD2BIA7M5AH3NBLDV7X73XANCNFSM6AAAAAAQDHGTDQ. You are receiving this because you were mentioned.Message ID: @.***>

paulpilone commented 1 year ago

This issue should be resolved by the work done in #280. That Issue addresses deleting and updating items which should help keep the total documents number in line.

paulineobps commented 1 year ago

Yes thanks the numbers are now in line.

Pauline Pauline SIMPSON Ocean Best Practices System Project Manager https://www.oceanbestpractices.org/

UNESCO /IOC Project Office for IODE, InnovOcean Campus – Jacobsenstraat 1 8400 Oostende, Belgium Email: @.**@.>; Alternate email: @.**@.>; Skype: Pauline Simpson ORCID: 0000-0003-2551-5740

http://www.iode.org

From: Paul Pilone @.> Sent: 19 May 2023 11:32 To: iodepo/OceanBestPractices @.> Cc: Simpson, Pauline @.>; Mention @.> Subject: Re: [iodepo/OceanBestPractices] new Search interface Stats display stuck at 1705 (Issue #257)

CAUTION: This email is external from UNESCO. Please be vigilant on its sender and content. ATTENTION : Cet e-mail est externe à l'UNESCO. Soyez vigilant sur son expéditeur et contenu.

This issue should be resolved by the work done in #280https://github.com/iodepo/OceanBestPractices/issues/280. That Issue addresses deleting and updating items which should help keep the total documents number in line.

— Reply to this email directly, view it on GitHubhttps://github.com/iodepo/OceanBestPractices/issues/257#issuecomment-1554829997, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFZQUQ7X23GTMRXFVX4TIGTXG6OBXANCNFSM6AAAAAAQDHGTDQ. You are receiving this because you were mentioned.Message ID: @.***>