Integration with WRAS - Githubissues

scottveirs commented 1 year ago

[x] Fall 2022: provide Acartia API access, documentation, and technical support to Ocean Wise and Lapis.
[x] Fall 2023: post Orca Network moderated sightings of key cetaceans in Washington to new Ocean Wise / WRAS API
[ ] 2024: post Orcasound moderated hearings of key cetaceans to WRAS API?
- [ ] No: Does OW want raw acoustic detections (how often and with what species and true positive confidence)?
- [ ] P1: Does OW want all moderated/confirmed acoustic detections, from both human or machine detection stream?
- [ ] Should the location of the presence record simply be the lat/lon of the hydrophone where the detection occurred?
[ ] 2024: post other details about cetacean sightings to WRAS API?
- [ ] P1: Post number of animals, at least for [Orca Network] records?
- [ ] P1: Add direction to POST after Cascadia web app has direction dropdown menu
- [ ] Invite WRAS to ingest all trusted observations within revised Acartia trust model?
- [ ] Map observer vs curator expertise, confidence, and trust level to WRAS species code 13 = Unidentified whale (and ecotype codes 00 Unknown, 01 Possible Resident, & 04 Possible Bigg’s (Transient)?)?
- [ ] Enhance attribution for WRAS end users by including comment field in modal, i.e. attribution back to individual observer, not just organization and sourceEntity?
[ ] 2024+: possibly automate an (annual? monthly?) exchange of Whale Report data from within Washington to Acartia...

Latest API documentation & tools:

Acartia: endpoint docs | contributing docs
Postman API query web app
WRAS API docs

Common species and their WRAS codes (via 11/2023 reference GET):

01 Killer whale
02 Humpback
03 Grey/gray
04 Minke
05 Fin
06 Sperm
13 Unidentified whale
14 Killer whale
15 Harbour porpoise
16 Dall's porpoise
17 Pacific white-sided dolphin
21 Common dolphin (short or long beaked)
22 Unidentified dolphin or porpoise

WRAS KW ecotype codes:

00 Unknown
01 Possible Resident
02 Northern Resident
03 Southern Resident
04 Possible Bigg’s (Transient)
05 Bigg’s (Transient)
06 Possible Offshore
07 Offshore

almitch commented 1 year ago

Thanks for you efforts @scottveirs and @aalaydrus - this is great. Glad that this is working as expected! We queried Lapis on a a separate issue we noticed with our test environment and turns out there was a bug in the API they have now fixed.

Not sure on the details but this may have been the cause of the previous GET request weirdness.

Either way, glad it is working as expected, and thanks a lot again for the work.

scottveirs commented 12 months ago

Hey @dylanse and @almitch -- Ali and I just submitted a TEST point and are satisfied with the performance of the AWS-based script that is currently posting new [Orca Network] sightings to the test server every 15 minutes.

Here is evidence that the date-time stamps look consistent to us:

1) Test point was submitted to Acartia at 8:22

2) Adding 7 hours now (NOTE: daylight savings kicks in on 11/5...), we see the Acartia record has a consistent creation date-time: "created":"2023-11-02 15:22:00" (8:22+7:00=15:22)

3) AWS Cloudwatch says that the script found and POSTed the new observation 2-3 minutes later at 2023-11-02T15:24:41

4) Which matches the minute and even second values in the WRAS dateReceived field (despite the local/Zulu bug we documented earlier) when we do a GET request to the WRAS test API using the old bearer token yielding this response:

            "dateReceived": "2023-11-02T08:24:41.000Z",
            "id": "18b55107-5ef3-44b8-a295-0a85bf6236ef",
            "behaviour": null,
            "comments": "[Orca Network] TEST not a real sighting (Scott Veirs)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": null,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -128.77306,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 46.09225,
            "speciesId": 2,
            "sightedDate": "2023-11-02T15:22:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "7dde6dbb-880b-4571-89b8-36f7dfd24645"

Most importantly, the "sightedDate": "2023-11-02T15:22:00.000Z", part of the response has the correct value in the ISO 8601 format that you specified.

I've double-checked that the latitude and longitude also match the initial record in Acartia.

Can you confirm this test POST looks good to you?

If so, we suggest continuing to let the automated script POST the test API today. If those points also look good to you we can try a brief switch to the production API tomorrow. (BTW -- I tested the new bearer token and a GET to the PROD endpoint and the request was successful.)

almitch commented 12 months ago

Hi @scottveirs - apologies for the late reply, I have been looking into the API and database most of today to try to confirm what is going on as I noticed some strange behavior. (@dylanse has been super busy today with other projects - he's in high demand! - so this spiel is also for his information)

So. All in all your POST looks good to me.

Not sure why the travelDirection field is autoing to 10, but on the front end this is displayed as ? - effectively a null - so we are getting the expected output. If we had some dev control I would love to just write a string detection logic to grab it from the comments, same with ecotype, but we'll work with what we've got for now.

Although your POST looks good, I have noticed some interesting bits in the visualization of the data sent through and in the database.

In a simple GET to review data in the test database I can see there are duplicates appearing from Acartia, but when I POST at my end there is no duplication.

An example of the duplication:

        {
            "dateReceived": "2023-11-01T17:13:13.000Z",
            "id": "04dca8d7-276b-441c-bbf6-c80ad0454ff2",
            "behaviour": null,
            "comments": "[Orca Network] Biggs T109A2s southbound 50 yards from Bush Point, moved offshore after passing point (Cindi Crowder Rausch)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": 0,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -122.61157,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 48.03067,
            "speciesId": 1,
            "sightedDate": "2023-11-02T00:05:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "59bc1246-b78e-4137-959b-1f159a732b18"
        },
        {
            "dateReceived": "2023-11-01T17:16:28.000Z",
            "id": "c573c45b-d811-426e-8c98-f613551ccb25",
            "behaviour": null,
            "comments": "[Orca Network] Biggs T109A2s southbound 50 yards from Bush Point, moved offshore after passing point (Cindi Crowder Rausch)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": 0,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -122.61157,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 48.03067,
            "speciesId": 1,
            "sightedDate": "2023-11-02T00:05:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "59bc1246-b78e-4137-959b-1f159a732b18"
        },
        {
            "dateReceived": "2023-11-01T17:14:15.000Z",
            "id": "bee36e1c-a65d-406e-8864-842470d0bc50",
            "behaviour": null,
            "comments": "[Orca Network] Biggs T109A2s southbound 50 yards from Bush Point, moved offshore after passing point (Cindi Crowder Rausch)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": 0,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -122.61157,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 48.03067,
            "speciesId": 1,
            "sightedDate": "2023-11-02T00:05:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "59bc1246-b78e-4137-959b-1f159a732b18"
        },

Before I ask Lapis about this, do you think this could be on your end that duplicates are being sent? You can see that the sightedDate remains the same between the detections but dateRecieved varies. Any thoughts?

This is also shown in the front-end test env for the app from alerts occurring yesterday:

Secondly, the front end map is visualizing the alerts as if the sightedDate is PST, not UTC. Take this example...

I submit this "POST"...

{
    "longitude": -123.21438868917642,
    "latitude": 49.30010974606236,
    "speciesId": 1,
    "sourceType": "observed",
    "idConfidence": 0,
    "travelDirection": 2,
    "sourceName": "app",
    "sourceEntity": "string",
    "externalID": "TEST-alex-4",
    "sightedDate": "2023-11-02T17:10:00.000Z"
}

and straight away an alert is generated, therefore it is treating the sightedDate as PST, rather than the UTC format it is in.

If I submit a UTC time now (+7) I would expect the alert to appear, it does not. Meaning, alerts are currently being visualized 7hrs in the future (soon to be 8). Looks like there might be a discrepancy between how Lapis are treating dates that needs to be fixed asap. I will write up a bug report to Lapis now and see if they can deal with it ASAP.

scottveirs commented 12 months ago

Hey @almitch -- Thanks for the updates.

We noticed the triple-record issue last week and Ali resolved it over the weekend. The latest test point we submitted was a successful verification that only a single record was posted by the de-bugged Lambda function. I'll be watching tomorrow to ensure we don't get more triplets, but we are 98% sure the issue is resolved.
I think we should table the direction question for now. We have plans to add it as an input field for Conserve.io apps and to eventually derive it for Orcasound acoustic detections. Either way, we elected to not parse it from the comments during this initial phase of API exchanges.
That is potentially a problematic bug for the WRAS front end! It seems similar (possibly related?) to the ambiguity previously mentioned in the dateReceived field. Maybe you should try an ISO 8601 format that indicates the time zone and see what the front end displays?

I'd suggest trying a format like 2023-11-02T19:44:40−07:00 ... and if it works for your test point, then maybe Ali and I send in the same format (after converting to PST of course) as an interim solution for [Orca Network] sightings?

almitch commented 11 months ago

Thanks @scottveirs -

Looks like there are no more tripled reports, thanks for checking in on this.
That's all fine, I was just thinking aloud on what we could potentially do our end in the future.
Both datetime fields are displaying UTC, but processing the times as PST.

So, what's the solution? The format you suggested is not accepted by the API, however, a work around for now could be to not include sightedDate on the POST. The database logic assumes if no sightedDate has been provided then the detection is live and presents the dateRecieved on the front end.

Although not accurate, I suggest we do this until we get the datetime fields functioning as they were intended to. What do you think? Is that easy enough your end?

scottveirs commented 11 months ago

With SRKWs coming into Puget Sound for the 4th time as I type, Alex @almitch , this sounds like a good hack to me. We will give it a try now.

almitch commented 11 months ago

Great, I'll be watching!

scottveirs commented 11 months ago

As an oceanographer it makes me feel a little ill manually subtracting 7 hours from a proper Acartia UTC timestamp (total javascript hack), but this humpback test looks promising @almitch --

Without the subtraction:

With the subtraction (after forcing the Lambda function to run between scheduled 15 minute events):

Next will be a test of a gray whale submitted via the scheduled run. If that works I'll try switching to PROD API and token...

almitch commented 11 months ago

Looks like both have worked great. Apologies for the ill provoking task! Hopefully Lapis can sort this out ASAP, I also highly dislike sticky taping anything, especially automated processes!

scottveirs commented 11 months ago

Hey @almitch -- It looks like the gray whale test POSTed with no errors from the Lambda function's perspective...

, but I don't see it with a GET request to the WRAS test API -- even after waiting for ~5-10 minutes.

Do you see any evidence of it?

scottveirs commented 11 months ago

Ok, I'll take that previous post as confirmation the gray whale test arrived.

Proceeding to switch to new endpoint and token, then await a real sighting for an end-to-end test!

scottveirs commented 11 months ago

Oh -- I see it now in the test API response to my GET, @almitch . I expected it to be at the top of the response, but found it further down:

Proceeding with swap...

almitch commented 11 months ago

This is from my GET request...

Just a little further down as the GET orders by time.

scottveirs commented 11 months ago

Ok swapped over to live WRAS endpoint, and I see this new report that should come in the next 15 minute cycle:

"created":"2023-11-03 23:35:00", ....
"data_source_comments":"[Orca Network] J pod at least southbound fast, spread out north-south and from midchannel to Whidbey (Cindi Crowder Rausch)

almitch commented 11 months ago

And there we go! That has gone through on the correct time as well! Excellent!

scottveirs commented 11 months ago

Ok, great! @aalaydrus or I will switch the hack to -8 at midnight on Saturday.

almitch commented 11 months ago

And from the GET on the prod API we can see the data:

"sightings": [
        {
            "dateReceived": "2023-11-03T16:47:46.000Z",
            "id": "32dd2fc9-c05d-4040-afec-e32e08efd5f1",
            "behaviour": null,
            "comments": "[Orca Network] J pod at least southbound fast, spread out north-south and from midchannel to Whidbey (Cindi Crowder Rausch)",
            "countMeasure": null,
            "disstressedState": null,
            "ecotype": 3,
            "experience": null,
            "howUHeard": null,
            "howUHeardSpecifics": null,
            "idConfidence": 0,
            "locationDesc": null,
            "longitude": -122.61686,
            "numberOfAnimals": null,
            "numberOfAnimalsMax": null,
            "numberOfAnimalsMin": null,
            "orgName": "Orca Network",
            "photos": null,
            "reporterEmail": null,
            "reporterLastName": null,
            "reporterName": null,
            "reporterPhone": null,
            "resiPostCode": null,
            "seaState": null,
            "sightingDistance": null,
            "sightingPlatform": null,
            "travelDirection": 10,
            "vesselName": null,
            "whenInWater": null,
            "whyInWater": null,
            "windSpeed": null,
            "latitude": 48.02038,
            "speciesId": 1,
            "sightedDate": "2023-11-03T16:35:00.000Z",
            "sourceType": "observed",
            "sourceName": "other",
            "sourceEntity": "Acartia",
            "externalID": "10c3993d-39ce-4d59-ab43-33f26b3c4e1a"
        },

This seems to have worked great, thank you so much Scott and Ali for the work!

I wonder if, given the SKRW are in the Sound, that we leave the API as is for the weekend, and revisit on Monday to make sure all is well? I'm happy to take your steer on this given the time change switch coming up!

scottveirs commented 11 months ago

Thanks for helping with an end-to-end test, Alex. I'm comfortable leaving it running as I'm confident we will be able to change the script that night (when there won't be any incoming sightings), so won't contaminate the production site with erroneous time stamps.

almitch commented 11 months ago

Fantastic, look forward to picking this up again on Monday! Thanks so much both, and have a brilliant weekend!

almitch commented 11 months ago

Hi @scottveirs, hope you're doing okay.

We've just had a meeting with our developers, we're clarifying the attributions in app and it should be done by the end of the week.

Sorry to add more work to your plate, but we're wondering if you could edit your POST slightly to ensure that it aligns with how we will also attribute hydrophone detections and Whale Report detections etc.

What this - could you please submit...

orgName: Acartia
sourceEntity: Orca Network via Whale Alert

(flipping the current submission of orgName and sourceEntity)

This way we do not have to hard code any values, and we're ensuring all organizations are following the same structure.

To confirm, following the data sharing agreement, does the above look okay in terms of attributing the correct orgs (I assume we would like to put Whale Alert on the info card to attribute them as well)?

Lastly, the website is updated with the current data providers, please do let me know what you think!

Thanks so much again!

scottveirs commented 11 months ago

@almitch,

I've just made the requested field flip. I've modified the sourceEntity value to Orca Network via Conserve.io apps.

This should clarify for WRAS end users that Orca Network and other Washington sightings providers are using not only Whale Alert mobile app for geo-referencing occurrence data, but also the Cascadia web app. Both apps were developed by Conserve.io. Eventually, another Conserve.io mobile app, Ocean Alert will also provide observations to the Acartia.io data cooperative...

@

scottveirs commented 11 months ago

Re attribution, @almitch -- from the perspective of Beam Reach, the current attribution on the WRAS web page looks good to me:

I'll start an email thread to get feedback from the other data cooperative members.

almitch commented 11 months ago

Thanks a lot @scottveirs - that's brill. Much appreciated.

The updates to the app should be ready for testing my OW Staff today, so hopefully they will be able to go live tomorrow or Wednesday.

I'll await your email, or look out for another GitHub post, RE the website attribution.

Cheers!

almitch commented 11 months ago

Hi @scottveirs - Lapis pushed an update to the API just now which fixes the timezone issue we were experiencing before.

When you get the chance could you please remove the "hack" you put on your POST and submit data in UTC (which will be displayed as PST on the front end). This will ensure that mariners are alerted at the correct time!

Cheers!

scottveirs commented 11 months ago

Hey @almitch - The hack has been removed. You should see next Orca Network posts in UTC now...

scottveirs commented 11 months ago

Confirming @almitch that a test point entered at 9:12:00 Pacific seems to have worked:

"id": "065b593f-bd2b-4ab7-8a18-e278d18b2bcf",
"dateReceived": "2023-12-02T17:17:46Z",
"sightedDate": "2023-12-02T17:12:00Z",

almitch commented 10 months ago

@scottveirs - thanks so much, that looks great. I appreciate the test point as well.

The attribution is now live as well.

scottveirs commented 3 months ago

Happy summer, @almitch and @dylan-oceanwise -- I may have just found a bug and/or ambiguity in the WRAS API.

I was trying to check on an Orca Network Acartia data point that WSF said didn't show up via WRAS a few days ago. I tried using the date range parameters in the API call, replicating the date-time format that is returned by the WRAS API from a GET request with no parameters, which PostMan formats like this:

https://sightingsapi.ocean.org/sightings?sightedFrom=2024-07-14T16:00:00Z&sightedTo=2024-07-14T19:00:00Z

Here's a screenshot of the query and WRAS API response:

The response appears to be what I'd expect if the sightedFrom and sightedTo parameters were submitted in the local/Pacific time zone (GMT-7 or Z-7), but I tried to specify my desired time range in GMT-0 with the trailing Z -- consistent with the date-time format of the WRAS API responses. (You've previously assured me that all response times really are in GMT-0 or Z.)

So, I think this suggests either that your API documentation should make the input and output date-time formats and time zones unambiguous (and ideally provide some syntax-perfect examples)...

...or, there is a bug in the API backend code which is assuming the input From/To times aren't already GMT-0 times and is therefore adding 7 hours to them (despite the user trying to specify they are already in GMT-0 with the trailing Z).

almitch commented 3 months ago

@scottveirs - thanks so much for the detailed description of this bug (which certaintly is a bug), I've been taking a look at this this morning.

We recently had to do a rollback on some changes to WRAS and updates to the API due to a deprecaited docker compose script that was found... Given the time of your comment it might have overlapped with our downtime due to this rollback. I'm also imagining this bug may have been introduced then.

You're totally right its erronously adding 7hrs. The database currently assumes all data is in GMT. It does not impact performance of WRAS, data is being logged and reported correctly, but spat out by the API incorrectly when using the filters.

I've raised a ticket with our devs to address this. Thanks for pointing this out.

If i've mentioned this before, ignore, but we are moving away from our current developers to a new local group, so, I'm hoping, there will not be any issues like this in the new year.

salish-sea / acartia

Integration with WRAS #16