Integration with WRAS

[scottveirs]

• [x] Fall 2022: provide Acartia API access, documentation, and technical support to Ocean Wise and Lapis.
• [x] Fall 2023: post Orca Network moderated sightings of key cetaceans in Washington to new Ocean Wise / WRAS API
• [ ] 2024: post other cetacean sightings and Orcasound moderated hearings of key cetaceans to WRAS API
  • [ ] Post number of animals, at least for [Orca Network] records?
  • [ ] Invite WRAS to ingest all trusted observations within revised Acartia trust model?
  • [ ] Map observer vs curator expertise, confidence, and trust level to WRAS species code 13 (and ecotype codes 0, 1, 4?)?
  • [ ] Add direction to POST after Cascadia web app has direction dropdown menu
  • [ ] Enhance attribution for WRAS end users by including comment field in modal, i.e. attribution back to individual observer, not just organization and sourceEntity?
• [ ] 2024+: possibly automate an (annual? monthly?) exchange of Whale Report data from within Washington to Acartia...

Latest API documentation & tools:

• Acartia: endpoint docs | contributing docs

• Postman API query web app

• WRAS API docs

Common species and their WRAS codes (via 11/2023 reference GET):

01 Killer whale
02 Humpback
03 Grey/gray
04 Minke
05 Fin
06 Sperm
13 Unidentified whale
14 Killer whale
15 Harbour porpoise
16 Dall's porpoise
17 Pacific white-sided dolphin
21 Common dolphin (short or long beaked)
22 Unidentified dolphin or porpoise

WRAS KW ecotype codes:

00 Unknown
01 Possible Resident
02 Northern Resident
03 Southern Resident
04 Possible Bigg’s (Transient)
05 Bigg’s (Transient)
06 Possible Offshore
07 Offshore

[scottveirs]

@dylanse This would be an optimal place for you to interact with Acartia folks like me and @aalaydrus as we aspire to push some Washington (WA) sighting records through the new WRAS API.

The initial goal is to send trusted visual sightings in WA from Orca Network across the border. Acoustic detections (raw or moderated) from Orcasound and OrcaHello AI will be a future consideration.

[scottveirs]

@dylanse Got any updates for us?

[dylanse]

@scottveirs The staging version of our API was just recently deployed to accept hydrophone detections. There is a meeting scheduled with our technical team next week to discuss the timeline for the readiness of the API to receive sightings data from Acartia. I anticipate your team should be able to start testing very soon.

For your reference, I can also share this API documentation with you to preview, https://oceanwise.stoplight.io/docs/sightings-network-api/80d0ea7ea0c45-create-a-sighting

This API and this documentation will serve both our autonomous and observed detections. Let me know if you have any questions, I will send you your API token soon as it's ready.

[scottveirs]

@dylanse Any updates on when tokens may be issued?

[scottveirs]

6/16 update from Dylan via email: Ocean Wise is working to "complete the GET route and provide us authentication keys." Expect an update by end of June...

[scottveirs]

7/12 update from Dylan via email: "our vendor has been making progress as of lately with the remaining POST endpoint on our API and the final testing version is to be delivered no later than next week. The necessary work to receive to hydrophone detections has been finished and it’s just a matter of adjusting some remaining fields to accommodate our observed sightings to the same endpoint."

@aalaydrus -- Dylan indicated that Acartia devs should be set up to begin testing soon. "access key and newest documentation will be provided next week"

[almitch]

Hey @scottveirs and @aalaydrus - I just wanted to follow-up on this as I'm taking a look at this issue for the first time in a while - thanks Scott for sharing the other day. Dylan followed up with documentation and the API key on the 21st of July first thing in the morning.

• Our updated API documentation
• The API key was included in the email, I won't share it here!

We are close to releasing a new production version of our applications, we'd love to include any changes we might need to make to accommodate Acartia sightings in this release if possible 😊.

Let either of us know if there's any questions! Thanks!

[scottveirs]

Thanks for following up during the busy summer season, @almitch . Looking forward to more progress this month...

Thanks for the API docs and key. Ali and I have discussed and he is working on test submissions of the requested real time sightings moderated by Orca Network.

@aalaydrus will seek answers here and/or via email. I'll provide other updates on scheduled calls tomorrow.

[almitch]

Thanks for following up during the busy summer season, @almitch . Looking forward to more progress this month...

Thanks for the API docs and key. Ali and I have discussed and he is working on test submissions of the requested real time sightings moderated by Orca Network.

@aalaydrus will seek answers here and/or via email. I'll provide other updates on scheduled calls tomorrow.

Great! Thanks so much Scott. Looking forward to our chats tomorrow.

Again, always here to help if you need anything.

[scottveirs]

@aalaydrus Based on our latest call with Alex @almitch these are the species we should push:

Southern Resident killer whales Bigg’s killer whales Humpback whales Minke whales Grey whales Fin whales

All other species or group IDs should be ignored for now.

As we've discussed before, to ensure only high-confidence observations are conveyed to commercial ships during this initial transboundary data exchange testing phase, the only records to be pushed should be those in which the Comment field begins with [Orca Network]

[scottveirs]

@almitch and @dylanse -- Is the mock/test API server functional with the token you provided? Ali @aalaydrus and I tried it today and it seemed down.

Or do you want us to test the production API server with a single Orca Network moderated sighting (before starting to create-sightings through regular automation)?

[dylanse]

@scottveirs, I know the developers were performing some migration changes today to prepare the production environment, I will report back once I confirm that the testing API is ready again.

[dylanse]

Hi @scottveirs and @aalaydrus, the dev work on the staging API has been completed, feel free to resume testing

[aalaydrus]

Hi @dylanse,

It seems that the both the live server as well as the mock server doesn’t seem to work through postman (with token)

However, I’ve discovered that the docs that they sent on their API seems to have a section to test the API call through the site itself here: https://oceanwise.stoplight.io/docs/sightings-network-api/1f68387dbeb27-get-sightings

Trying through the above site to do a GET request on the live server and the mock server also gave me the same set of errors

I tried the POST on the mock server and I seem to also get a similar error message which seems to be outputted from Prism

[dylanse]

@aalaydrus The staging server appears to be operational from my end. However i just noticed that there was a {basepath} appended to the url that shouldn't be there. I have removed it from the documentation.

The mock server API is provided out of the box by the documentation provider but it's not something we use or rely on for any testing use.

Hope that helps, look forward to seeing some test data coming in soon!

[scottveirs]

Hey @dylanse and @almitch -- we'll keep testing this weekend and next week, but in the interim (and maybe to accelerate our progress) can you answer these two queries?

1. Have any other external entities (outside of Ocean Wise and Lapis) succeeded in pushing records via your new API?
2. If so, could you put us in touch with any of them so we could compare notes/scripts/approaches?

I'm facilitating a hackathon this Mon-Wed with Microsoft. One goal is to start pushing OrcaHello data (automated ML detections moderated by bioacoustic experts) from Azure to Acartia. So, I'm particularly curious to know if any Canadian hydrophone hosts have started to add acoustic detections to WRAS via the API.

Thanks, Scott

[dylanse]

Hi @scottveirs

1. Yes, we have had live hydrophone data streaming successfully to the API since June 2023. You can also see this data by performing a GET request for the type: autonomous (see API documentation)
2. That is a possibility but I'd like to provide support from our end first. I'm happy to get into a chat/call with @aalaydrus and help diagnose what the issue is as we don't see any anything precluding a successful connection

[aalaydrus]

Hi @dylanse finally got both GET and POST to work from my end. Took me some time to figure out that the token field it was expecting was x-api-key as I assumed that the API url is sitting behind an API Gateway in AWS and it is what is expected to authorise requests.

I'll communicate with @scottveirs for entering the other data and other pending tasks.

Just a few questions for you on pushing our data into the WRAS API @dylanse:

1. We noticed that in the Observed data schema, there were multiple Killer whale values in the speciesId field. Should we use 1 or 14 if we want to indicate a killer whale of any ecotype.

2. Some of our fields for the species, are already named or valued as "Southern Resident Killer Whale". Would this mean if we push this data that we need to have speciesId=1 and ecotype=3

[scottveirs]

Hey @dylanse -- can you provide answers to these two questions ASAP ^^^ this week?

With them we should be able to start POSTing any new observations of any of the target species every 5 minutes...

[dylanse]

Hi @aalaydrus,

1. Thanks for spotting the duplicate Killer Whale value (14) in our speciesId field. I have made a note to correct this but will need to verify it's not there for backward compatibility first. Please proceed and use 1 as the value for Killer Whales.
2. Correct, a Southern Resident would be speciesId=1 and ecotype=3

Also, something new but not documented yet is that there is a /reference endpoint you can hit to access the possible values programmatically if needed.

[scottveirs]

Hey @dylanse and @almitch

A couple questions and comments as @aalaydrus and I deploy the transfer script to AWS this week:

1. Your API documentation could be improved by verbally describing/defining the dateReceived and id fields (beyond specifying their format as string and uuid respectively). Are we correct in assuming these fields will be populated upon successful post of an Acartia observation, so we should leave them as null values?
2. Your API documentation is confusing (to Ali initially, and now also to me as I study it more closely) because the landing page at https://oceanwise.stoplight.io/docs/sightings-network-api/80d0ea7ea0c45-create-a-sighting defaults to the Autonomous data scheme, when we were asked to post Acartia observations (Orca Network sightings, presumably via your Observed data scheme). Additionally, when the dropdown is used to change the API documentation to Observed the API examples do not update (Send API Request button code and curl example code stay the same; compare 2 screenshots below). You probably made this UI default/decision to accommodate Canadian providers of automated acoustic detection algorithms. Could you confirm that we're interpreting the situation correctly, and if so could you reduce confusion for future visual observation providers, e.g. by adding explanatory text to the top of the WRAS API guidance?
3. Related to 2 (and in anticipation of some day adding Orcasound automated acoustic detections), in your default example API query -- for what appears to be an automated click detector (detectorName = SpectroDetector-v3.5.1...) -- shouldn't the sourceType on line 20 be equal to "Autonomous"? Instead it seems that the default was assumed -- as "sourceType": "Observed"... (see first screenshot below.)

[almitch]

Thanks Scott, this is really useful feedback, especially given that this API should be general enough for several types of data from different orgs so it's great to have some input into it. Thanks.

Dylan and I have a meeting tomorrow to discuss ongoing integration work so we can work on some ways to improve the clarity of the documentation.

I'll leave @dylanse to get back to you on the more specific questions/

Cheers.

[scottveirs]

You're welcome, @almitch . When you and @dylanse meet, it might be efficient to give also us a yes/no or comments/concerns regarding any fields and/or values that we're test-posting to the WRAS API:

{
  "longitude": -124.06493,
  "latitude": 49.39968,
  "speciesId": 1, // 1, 2, 3, 4, or 5 
  "sightedDate": "2023-02-02T06:39:32.000Z",
  "ecotype": null, // unless KW: then 3 if SRKW, and 0 otherwise
  "comments": "The full Acartia comment string.",
  "idConfidence": 0,
  "orgName": "Orca Network",
  "sourceType": "Observed",
  "sourceName": "other",
  "sourceEntity": "Acartia",
  "externalID": "a unique string equal to the record identifier in Acartia",
}

Note that we are not providing the following fields (intentionally leaving them as nulls in the posted record):

dateReceived
id
countMeasure
numberOfAnimals

The last we could provide in the future, but only with an ambiguous value for countMeasure...

We're eager to finalize these last details and automate posts via AWS Lambda/Eventbridge as soon as we have answers in hand!

[scottveirs]

@dylanse Here is are two more tiny bits of feedback about the WRAS API based on our successful test posts (via axios.post in Ali's Node.js script):

A) Upon getting one of our test records, we noticed that the travelDirection field had been populated with a value of 10 when we did not include that field in our POST. Is that a bug on your end, or an intentional feature?

B) In the example API Request there seems to be an errant final row (see screenshot below). Perhaps this is leftover from creating the example?

[scottveirs]

@almitch Alex -- any new updates from the OW IT team before we start posting?

[dylanse]

Hi @aalaydrus @scottveirs, thank you for your feedback on our API documentation.

I've taken steps to revise and clarify the previously mentioned ambiguities in our documentation. The id is a UUID that can be generated and included with the sighting. If this field is missing, a new UUID will be generated and returned to you. The dateReceived field is automatically populated in both autonomous and observed responses, indicating when the sighting was logged on our end.

As part of the documentation we are currently using, it has a component that automatically auto-generates example requests and responses. Unfortunately, this doesn't work perfectly and should not be relied upon. It would be my preference to remove that component altogether but did not see a way to remove it last time I checked. Apologies for any confusion this has caused.

Your example request appears to be in good order, so feel free to proceed whenever you're satisfied with the explanations provided. Once we see your data coming in, we'll take a look and send any necessary feedback as soon as possible.

cc @almitch

[aalaydrus]

Hi @dylanse I'm getting the following error from the live server

error: { length: 85, name: 'error', severity: 'FATAL', code: '53300', file: 'proc.c', line: '360', routine: 'InitProcess' }

I was hoping to ask if there's a maximum amount of requests at a specific period to interact with the server. I am also getting this from a simple GET request from Postman. Thanks!

[scottveirs]

@dylanse Any thoughts on this error we’re getting? Ali has continued trying, but to no avail…Thus, we will have another call tomorrow at 8am to troubleshoot, so even preliminary thoughts from you and/or @almitch would be welcome.

Is the WRAS server refusing requests from our token, either because it’s expired or maybe due to too many requests? How often is JASCO posting acoustic detections? (Nothing wrong with our plan to post new qualifying observations as frequently as once every 5 minutes, right?)

[dylanse]

This looks like an error on the postgres side. I have no direct access to those logs so will have to get the devs to take a look. However, I do see several Acartia sightings when I perform a GET for observed sightings on the API (with your token). If it does not work for you, there could be some throttling going on if there if you attempted to send a large amount of sightings. I will confirm tomorrow and report back soon as I can

[scottveirs]

@dylanse Thank you for the update. We are standing by.

In the mean time, can you please confirm that posting "whale sightings" as often as once every 5 minutes will not be a problem?

If it will be a problem, please let us know how often we should run the post script.

[scottveirs]

@dylanse Out of curiosity and to get a point of comparison, how often is JASCO posting acoustic detections to the autonomous API?

[scottveirs]

@dylanse Is there a most basic no-auth request we can make of the mock and/or live WRAS APIs that would allow us to assess if it is responsive (prior to engaging in additional complexities of bearer tokens and authentication)?

I'm imagining you might have something like Acartia's no-auth endpoint --

https://acartia.io/api/v1/sightings/current

?

[scottveirs]

@dylanse Quick report after call this eve with Ali:

Glad to report that basic GETs with our bearer token again seem to be working as of an hour or so ago...

and BTW --

https://6671737ff1.execute-api.us-west-2.amazonaws.com/v1/reference

is helpful!

[dylanse]

Hi @scottveirs and @aalaydrus, I contacted the development team yesterday regarding the issue you experienced. They are currently investigating it and believe that the problem may stem from their existing configuration being unable to accommodate the high volume of requests.

JASCO currently sends a relatively low volume; this is the first time we've encountered an error related to a high number of requests.

For this initial phase, our API requires a token for access for all endpoints.

In the future, I'm happy to jump on a call/chat if needed to expedite any troubleshooting, feel free to drop me an email!

[scottveirs]

@dylanse Thanks for the update. In an effort to reduce the volume of requests, as of 2023-10-12 18:25:50 (UTC-07:00) we are posting only new Orca Network observations every 10 minutes to https://6671737ff1.execute-api.us-west-2.amazonaws.com/v1/sightings

You should see some TEST points I generated. I've confirmed at least a couple were successfully posted this evening, by checking via a GET.

Can you confirm receipt of the TEST point with Sighted Date = 2023-10-13T01:11:00.000Z and okay the formatting?

BTW: We still don't understand why the travelDirection value is getting set to 10 when we only provide a null.

Under the assumption that you don't test for duplicates (and therefore do not want all qualifying points we GET via https://acartia.io/api/v1/sightings/current ), @aalaydrus wrote custom code to implement our initial low volume scheme: posting only recent observations (with age less than 10 minutes). We will likely need to revisit this age threshold in consultation with your devs (to avoid hitting any high volume caps) and the data providers (who may require more or less time to submit a new observation).

Assuming you also receive the suite of new points that will likely be submitted tomorrow during "live" data entry by Orca Network and/or us, we hope you can proceed with any QA/QC and pass points ASAP on to WRAS end-users. (SRKWs, Bigg's, and humpback sightings were abundant today and will likely be so tomorrow as well).

[scottveirs]

@dylanse FYI: Randomly checked /reference endpoint this eve and got this error:

LMK if there is some other place y'all want such bug reports and outage notes...

[aalaydrus]

Heya @dylanse We've encountered the following error message when the we tried posting GET and POST requests:

{
    "message": "Internal server error"
}

Not sure if this has to do with our token or if the server is currently set offline.

[scottveirs]

@dylanse Just tried a simple GET and we are still getting an Internal server error message.

Can you give us an ETA on when this might be fixed? I am eager to continue testing the performance of our deployed POST script this week!

[dylanse]

@scottveirs @aalaydrus I can confirm that I am getting the same error. I have a meeting with the devs in 2 hours, will report back shortly!

[scottveirs]

@dylanse Perhaps it would be helpful to share with the Lapis folks that we are attempting a post every 10 minutes (for the last 9 days or so)?

[scottveirs]

@dylanse and @almitch Over the weekend that I noticed what I think may be an error in the WRAS API dateReceived field's time formatting that may be important to rectify sooner than later.

It appears your time stamp has a value that is accurate in local time, but the format with the Z suffix is indicating the wrong time zone. (Z implies Zulu aka GMT, UTC, or 000Z, whereas for the time value of the field to remain accurate the time zone should be Pacific or -700Z currently, or equivalent.)

My strong evidence from recent tests for this claim:

Step 1) Orca Network observation occurs: 2023-10-13 11:53:00

Step 2) Acartia record is created: 2023-10-13 11:54:30

Step 3) Ali's script posted it to the WRAS API using this GMT date-time stamp: 2023-10-13 18:55:48 GMT or 11:55:48 local

Here is the log entry for that POST:

  comments: '[Orca Network] Js, Ks, and some Ls. Many have stalled off Maxwelton, others spread to the north (Rachel Haight, Jim Pasola) ',
  ecotype: 3,
  idConfidence: 0,
  longitude: -122.47444,
  orgName: 'Orca Network',
  latitude: 47.9359,
  speciesId: 1,
  sightedDate: '2023-10-13 18:53:00',
  sourceType: 'observed',
  sourceName: 'other',
  sourceEntity: 'Acartia',
  externalID: 'e5a7faf0-58c2-40c1-bf18-a00da2c1c921'
}

Step 4) WRAS record is created (based on a GET): 2023-10-13T11:56:49.000Z

The date-time stamp in Step 4 is based on and copy/pasted from a GET to https://6671737ff1.execute-api.us-west-2.amazonaws.com/v1/sightings. The returned time value is reasonable if the associated time zone is local. However, the time zone is designated by .000Z which seems to imply a non-local time zone.

[dylanse]

Hi @scottveirs and @aalaydrus, just confirming for you that the API is operational again. The reason for the error was identified with a function that was not closing the db connections.

I will follow up again on the timezone issue you raised. Your detailed feedback here is greatly appreciated, thank you!

[scottveirs]

@dylanse -- Can you confirm/deny that successful posts are being transmitted to WRAS end-users at this point? Or will we need a new bearer token for a production version of your API?

[scottveirs]

Also @dylanse or @almitch -- based on feedback from Orca Network staff today, we will be posting every 15 minutes, rather than every 10 minutes. Related to this, it would be valuable to eventually get answers to these two question from you:

1. How old is "too old" for an observation to be useful (i.e. worth POSTing to the API) from the perspective of WRAS end-users?

2. How often is "too often" (aka redundant) for observations of the same individual or group? (In other words, if Orca Network is observing the same species/ecotype every 2 minutes, will each observation be shared via each of the WRAS notification methods? If not, what subset is shared via which notification method?)

[almitch]

Hey @scottveirs - apologies for the delay, I've got email notifications turned on now so I should be responding a bit quicker.

In answer to your questions...

• Posts are not currently being transmitted to end users as we need to get the acknowledgements sorted out in the alerts. We are waiting to hear back from Lapis on a quote and estimated delivery time for that. It is unfortunate we can't do this ourselves as it is a very easy thing to do but we are relying on Lapis to do the work in their time frame. You will need a new token for the production API (@dylanse correct me if I'm wrong on the token).

• Too old, 15mins is good for us, from conversations with mariners the more 'real-time' the better, extrapolating from that 20 mins and above it likely getting towards the "too old" spectrum, particularily for Killer Whales. The last thing mariners want is to recieve an alert and to be on top of the animals before they know it (worst case).

• This is an interesting question. Does Orca Network often report that frequently? For our sightings network reports we do not have any aggregation functions so multiple sightings in the same area are sent as multiple alerts. However, for our Hydrophone alerts via JASCO these are grouped until there is 30min of silence (if you see an alert from the hydrophone you know they v.likely are in the area). If Orca Network do record same species/ecotypes that often perhaps we will have to look at grouping these sightings together over a certain period, or, only alerting every say 30mins if the prev alert was the same species/ecotype + group size for instance.

Thanks Scott.

[scottveirs]

Thanks for the updates, Alex @almitch !

It is still relatively rare for Orca Network and other dense US sighting networks to generate unique observations more than every half hour. However, in the past two years there been many high-visibility, pleasant-weather days when a popular pod or matriline was tracked with unique locations reported every 5-10 minutes. I suspect Ocean Wise could delay adding the type of intelligence you describe for visual sightings this year, but would suggest you plan for it.

On a related topic, raw detections from Orcasound hydrophones by humans or machines are grouped temporally into "candidates." They are currently added manually to Acartia, but once moderated candidates are added via the Acartia API, then a similar issue to the JASCO situation may arise for WRAS end-users.

[dylan-oceanwise]

Hi @aalaydrus , @scottveirs , just following up on the timezone issue. Would you be able to convert and send your timestamps in in ISO 8601 format and we can always assume they are sent in UTC?

[scottveirs]

Hey @dylanse -- to be clear the date-times we are submitting are sent in UTC. (All Acartia timestamps are in GMT aka UTC aka Zulu time zone.)

We will try (briefly) to write more custom javascript to meet your request, but we don't have the budget to buy full ISO specs, so could you please just confirm you want the format you currently provide as an example in the WRAS API docs under Observed data scheme's sightedDate field?

2023-02-02T06:39:32.000Z

[scottveirs]

Hey again @dylanse Dylan, I was just looking to see if we could again use the WRAS API to test for successful POST using GET, but I'm not seeing data more recent than 10/13/23. Before we start hacking again, please answer both of these queries:

1. Can you confirm that we will be able to again use a GET as before to verify that new WRAS records have been created?

These are the most recent (still tripled, BTW) records we receive with our bearer token:

2. Can you check WRAS API logs to see if this most-recent POST made it through?

2023-10-26T19:50:17.596Z    a5653ac2-d186-454b-af72-e27fb07ee1f5    INFO    {
  comments: '[Orca Network] Jpod spread out milling with directional changes (Rachel Haight)',
  ecotype: 3,
  idConfidence: 0,
  longitude: -122.41083,
  orgName: 'Orca Network',
  latitude: 47.88697,
  speciesId: 1,
  sightedDate: '2023-10-26 19:45:00',
  sourceType: 'observed',
  sourceName: 'other',
  sourceEntity: 'Acartia',
  externalID: 'f2cfb92d-5fcc-4327-a73a-b24731af0122'
}

[scottveirs]

Hey @dylanse -- You can ignore the previous post. Ali and I just met and were able to confirm that his efforts yesterday to meet your ISO8601 request were successful. I'm not sure what was going on with my previous GET requests, but this morning I successfully retrieved this test point...

...and can confirm that the UTC date-time stamp in the WRAS record (2023-10-27T16:27:00.000Z)is consistent with the local time of the test observation (9:27 am Pacific time, currently -07:00Z).