myrmoteras
commented
1 year ago add a page on Webhook/microservie https://github.com/plazi/arcadia-project/blob/master/webhook.md
Closed myrmoteras closed 1 year ago
myrmoteras
commented
1 year ago add a page on Webhook/microservie https://github.com/plazi/arcadia-project/blob/master/webhook.md
EXTERNAL SENDER
Hi Alex, So that means addition to that community will automatically send a callback now? Yes, there are 2 events that can be fired as we tested in Sandbox, one for when a record is requested to be added, and one for when it is accepted by a curator (this is the one you probably care more about). Is the JSON the format we defined last year, or do I have to adjust the parsing routine to the new format? Yes, the format is the same, some issues with generated links for the "files" key were also fixed (so you can i.e. download the PDF using the provided links and process it). Also, is there a place in the web UI settings or somewhere that I could modify the web hook (replace token, etc.)? Unfortunately, not at the time, we're adding this on our side via configuration. I've used the URL and access token that you shared last time, but if there are any changes needed just let me know and I'll update things ASAP. I now played the full user story:
Still not even a whiff of a callback in the Frankfurt Apache logs ... there must be something going wrong ...
If you still have the time today, let's just do this. I'm finishing up the Arcadia report as well, so given that we'll be testing on production this time, we'll probably need to be careful with the submissions. I will ping you on Skype once ready to give things a try. Cheers, Alex On 8/31/21 4:32 PM, Guido Sautter wrote: Hi Alex, Back from holidays, and from what I got we have to fast-track the webhooks use case for the final Arcadia report. @Donat let me know if there's anything to contribute in terms of reporting. As mentioned in the past the identifier name "biosyslitcontrib" doesn't matter much, since it only matters for the community's URL and for API usage. The human-friendly community name can change so we can go with "BLR Contributions" for the time being, and change/market differently in the future. great, thanks for this ... and sorry to not get back to you right away yesterday ... was in the middle of a decoding frenzy with Marcus and the other POA people to meet the numbers (still going on, actually).
I've created the community here under the same user that manages the "biosyslit" community, i.e. the one with the admin_coviho@plazi.org email. This is the email address that will be receiving also notifications for any users that request records to be added to it, so @Marcus (or anyone else "curating") should monitor this or just review the community's curation page every once in a while. So that means addition to that community will automatically send a callback now?
@Guido, would it make sense to organize a call for today to test everything on production? I've configured webhooks same way we tested in the past for the "biosyslitcontrib" community. Thanks, that's great. Is the JSON the format we defined last year, or do I have to adjust the parsing routine to the new format? Also, is there a place in the web UI settings or somewhere that I could modify the web hook (replace token, etc.)?
If you still have the time today, let's just do this.
Best, Guido
On 7/27/21 7:58 PM, Guido Sautter wrote:
Hi Alex,
I think the should be OK naming the BLR intake community "BLR Contributions", maybe with ID "biosyslitcontrib". Let's set it up this way and see if we get the whole thing to work, OK?
Best, Guido (Marcus) Just wondering: can I get a notification when a new item is added to the Ingest community? Yes, it would be possible in this new Ingest community, at the admin_coviho@plazi.org email, which is currently the admin account for both the "biosyslit" and "coviho" community. It is also the account that all Zenodo API tokens used by Guido, Marcus, Pensoft, etc. belong to. For the "biosyslit" community we have disabled email notifications (since it's part of the automated workflows, so you would have already received ~1million email notifications by this time). (Donat) Just to understand: do we need to start a new community, or we just need another username/login? We need to start a new community, named "Biodiversity Literature Intake" (or whatever name + ID we decide for) under the account mentioned above. Let me know if the above makes sense. In general having a separate email for owning the community and notifications is not possible at the moment. Best, Alex On 22/07/2021 00.00, Guido Sautter wrote:
Hi Donat, ? If so, Donat, could we have a specific login for this Ingest community?
Just to understand: do we need to start a new community, or we just need another username/login? Alex and me were actually thinking about a wholly new community, associated to BLR by name (how about "Biodiversity Literature Intake"?), to specifically act as a staging area for uploads that are meant to be ingested by Plazi ...
Once processing is done, they will go into BLR proper from the uploader inside our server.
The point is to prevent taking in really everything added to BLR by other means than the uploader inside our server, e.g. non-biodiversity depositions made by Pensoft.
If the latter tell me and I will provide you a username, a dedicated email. It is the former, but we'll still need a dedicated email, alongside a name, which the question originally was about.
Cheers, Guido
From: Guido Sautter gsautter@gmail.com Sent: Wednesday, July 21, 2021 9:02 PM To: Marcus Guidoti marcus.guidoti@gmail.com; Alex Ioannidis a.ioannidis@cern.ch Cc: Donat Agosti myrmoteras@gmail.com; Terry Catapano catapanoth@gmail.com Subject: Re: Zenodo Webhooks - Suggestions Wanted
EXTERNAL SENDER
Hi Marcus, I'm glad you guys went for this ingest community idea. I was a bit concerned about leaving the real BLR open to submissions of anyone, without any kind of curation. there was never an idea to do anything without curation ... even the "ingest" community will need curation, but we still need it as sort of a gateway anyway, and as a curation stage, as otherwise we'd also ingest any PDFs Pensoft puts into BLR ...
Just wondering: can I get a notification when a new item is added to the Ingest community? That's for Alex to answer.
I think it will be useful to trigger internal processes of task creation and distribution so more people, not only me and Donat, keep an eye on this. It will surely be more effective if we can spread the workload. If this is possible, I think we all will have to share the same login, right? If so, Donat, could we have a specific login for this Ingest community? And that's for Donat to answer.
Also, please give me your opinions about naming the "gateway" community ... we need this.
Best, Guido
Em qua., 21 de jul. de 2021 às 12:34, Alex Ioannidis a.ioannidis@cern.ch escreveu: Hi all, (@Donat let's continue here our abruptly cut Zoom chat). As Guido mentioned, we've ironed out most of the rough spots of the webhook-based workflows, with the exception of the Pensoft originated uploads. From the proposed solutions, having a separate community configured with webhook, might be the way to circumvent this issue (and also keep our uploads separated). I'm in favor of creating a dedicated "BLR Community Ingest" Zenodo community (with the id "biosyslit_community_ingest"). I think once we have consensus on the naming (since it won't be easy to change the community ID later on), we could launch our first tests and possibly even get a "controlled" internal use-case be processed end-to-end. Let me know what you think, and if we should have a call one of these days to discuss in more detail. Cheers, Alex On 16/07/2021 17.26, Guido Sautter wrote:
Hi all,
I'm happy to tell you that Alex and me just successfully tested the webhook notification triggered by a person with the credentials adding an article deposition to the BLR community via the Zenodo web UI ... deployment of the feature to the live system is scheduled to happen next week.
One thing we came across is this, though: How to prevent articles added to BLR from the outside, e.g. PDFs uploaded by Pensoft? Additions coming from our own server are covered and caught, but additions from Pensoft are an issue, as we don't necessarily intend to ingest those ... after all, we already import the TaxPub versions ...
A possible solution we came up with is a secondary community, as sort of a staging area ... once an article gets added to that staging/intake community by a curator (e.g. Donat or Marcus), it gets ingested and then flows back to the BLR community proper, together with all the extracted data, just like used to. We both think this solution makes lots of sense, and is fairly easy to implement. Just one question remains: What to name this staging/intake community? It's the entry point for community contributions, after all, so the name kind of does matter in terms of acceptance ... Here a few ideas:
My preference is on the first one ("biosyslit_contribute"), but you might have other ideas ... please let me know, as this is the last missing piece to get this mechanism rolling.
Best, Guido
-- M.