Process Saildrone Data

[jrenfree]

Items needed for processing Saildrone data:

• [x] Template EV file
• [x] Template ECS file
• [ ] Interactive map of Lasker and LM CTD data
• [x] Interactive map of Saildrone and CTD data
• [x] Use 'processCTD-SH.R' to process the 2023 Shimad CTD data but using each of the Saildrone ECS templates to create ECS files for the Saildrone transects.

[jrenfree]

@kstierhoff do you think you could get an interactive map of the Saildrone transects along with all of the CTD casts from each of the vessels?

[kstierhoff]

Yeah, for (my) future reference, this is done using plotSaildrone, and I will try to get this working tomorrow.

[alicebeittel]

Thank you for this thread and preparation work!

I can work on making a new processCTD.R script tomorrow or Friday. I may ask one of you to look at it to make sure it looks good. I'm expecting to have to change some file paths such as pointing to the correct ecs.template for each saildrone vehicle.

[jrenfree]

Sounds good! Once it's ready I can help as well and we can process data in parallel, since it will have to be done three times and may take a bit of time.

[kstierhoff]

I'm updating plotSaildrone, which we can add the CTD/UCTD locations to and use to select ECS files for Echoview processing. The Saildrone position data in ERDDAP has gaps and doesn't appear to be updating, so I hope we can get that sorted out soon, but we should be good to go once we have cast locations and updated nav data from SDs.

[jrenfree]

Hi Alice,

I've been processing the Shimada CTD data to create Saildrone ECS files, which I think I'll just go ahead and run. I've already completed one of the Saildrones. So you don't need to worry about trying to tackle this.

[alicebeittel]

Fantastic. Thank you Josiah!

[kstierhoff]

I put a draft version of plotSaildrone here, and pushed changes to Github: \swc-storage4-s\AST4\SURVEYS\20230703_SAILDRONE_SummerCPS

Saildrone data folks are looking at the missing positions and we can rerun this once the full data set is available. Also, we can add any/all CTD/UCTD casts to make it easy to choose ECS files.

[jrenfree]

I've finished creating ECS files for each of the Saildrones using both Shimada and Lisa Marie CTD casts. So once we get the latest Saildrone nav data it would be nice to include the cast locations for both vessels.

[alicebeittel]

Woo!!! Do you have time to do the LBC too? Otherwise I can do tomorrow!

[kstierhoff]

Ok, so I put the latest version of plotSaildrone on AST4: \\swc-storage4-s\AST4\SURVEYS\20230703_SAILDRONE_SummerCPS

it now contains the Shimada (red) and LM (green) cast locations. I'll run one last time if they fix the SD nav data on ERDDAP.

[jrenfree]

Just finished processing LBC as well!

[alicebeittel]

SWEET! Thank you all so much! Kevin, do you think you could also add LBC casts?

[kstierhoff]

Absolutely, but I don't think I have a file containing cast locations for LBC. Also, I'm not sure the LBC data will help much with Saildrone processing, will it?

[alicebeittel]

I guess not since we have RL casts in those locations and those will be deeper and farther offshore than LBC casts.

Also -- We need to do a post-mission calibration? See below the most recent thread from Christina from the mission portal.

[kstierhoff]

SD routinely conducts a post-mission calibration, and Josiah scrutinizes the results. I let them know that Josiah would remain our POC, which may impose some delays.

[alicebeittel]

What does that mean for data processing? Can I still process the data or does this mean that I have to wait until we have completed a post-mission calibration analysis?

[jrenfree]

You can proceed with processing. The pre-survey calibrations looked good enough. Saildrone just likes to cross their T's, and if I ultimately find that one of the instruments looks bad in a post-survey calibration then we can go back and assess that data.

[alicebeittel]

Update: Processing for Saildrone 1048 is complete. Saildrone 1060 data is currently being downloaded to NAS1. We are waiting on Saildrone 1096 to be available on Saildrone's FTP.

[jrenfree]

SD-1060 data is now all downloaded and ready to be processed. It's currently on NAS1, however if I get confirmation that the AST4 backup is complete then I move transfer it onto there since that's where the SD-1048 data already lives.

[kstierhoff]

From Saildrone: The .RAW echosounder files, the GPS fix files, and the 1hz and 20hz netCDF files are now available for SD-1096 on Saildrone's FTP. All data from this year's mission has been delivered.

[alicebeittel]

Saildrone processing for 1048 and 1060 is complete. Both EV exports and nasc.r processing. I have uploaded the EV files, EV lines, CSVs, and Exported Images to AST 4.

[jrenfree]

Hey Alice, do you recall if you used any of the CTD casts from the hake survey on Shimada for processing the Saildrone data (i.e., ECS files)? Just wondering if I need to include that info in the survey report.

Thanks!

[jrenfree]

@alicebeittel @dmurfin @jpzwolinski: The last of the Saildrone data (SD-1096) has been downloaded, and all data transferred onto the AST4 drive. So this last bit of remaining data will need to be processed.

I don't recall who currently has the dongles, but I believe it's the three of you. I'd be happy to process some of the data myself, otherwise I'm hoping to finish up the survey report and get that out for review.

Lastly, Saildrone data still exists on NAS1, however please ONLY work with the data on AST4. NAS1 is a duplicate copy that I will ultimately delete.

[alicebeittel]

Yes! I have an EV license dongle. I will work on addressing the issues for the other saildrone lines and pick up processing 1096 this week. @dmurfin do you want to take the southern most chunk of 1096 and I'll start from the northern most chunk of 1096 lines?

Good copy on working only from AST4. It looks like there are still files that I completed and uploaded to NAS1 before Thanksgiving week missing from AST4. Are you planning to copy over what is on NAS1 to AST4?

Yes, I have only been using Shimada hake CTD casts for processing Saildrone data.

[jrenfree]

Sounds good, thanks!

Could you point me to where those un-copied NAS1 files are? I ran a SyncBack profile to compare NAS1 to AST4 and it found no differences, but I may be looking in the wrong place.

[alicebeittel]

Nevermind! I just took a look again and realized I was looking at the wrong window. I saved all my processing I did before Thanksgiving to AST4.

[alicebeittel]

@kstierhoff There is still an issue with the 1096 gps data in the PlotSaildrone file on AST 4...\SURVEYS\20230703_SAILDRONE_SummerCPS\plotSaildrone.html. I think we were waiting until we got the latest GPS data. Can you re-run plotSaildrone with the latest GPS data provided by SD for 1096?

[jrenfree]

I just realized that I never created the Echoview-compatible GPS files for SD-1096. I'll do that now.

[jrenfree]

OK done!

[alicebeittel]

Should we keep all of this transect 132, even though it is extremely zig-zag? Yellow highlighted area is what I would definitely exclude, unsure about the rest. @kstierhoff @jrenfree

[jrenfree]

I would do it how you have it. That is, keep all the zig-zags but exclude the last part you highlighted yellow.

On Fri, Dec 1, 2023 at 12:04 PM Alice Beittel @.***> wrote:

Should we keep all of this transect 132, even though it is extremely zig-zag? Yellow highlighted area is what I would definitely exclude, unsure about the rest.

image.png (view on web) https://github.com/nmfs-swfsc-ast/ast-tasks/assets/112714266/f4b6da9d-51c9-4b0d-ab47-44ce169870d5

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/52#issuecomment-1836707124, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACD2JYBDLLIXW2WTMLHWXYLYHIZ4ZAVCNFSM6AAAAAA5KBSTOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZWG4YDOMJSGQ . You are receiving this because you were assigned.Message ID: @.***>

-- Acoustic Engineer Advanced Survey Technologies Southwest Fisheries Science Center / NOAA 8901 La Jolla Shores Drive La Jolla, CA 92037 WORK: (858) 546-5669 CELL: (858) 232-3121

[jrenfree]

Finished with my small subsections of SD-1096 transects, however the spreadsheet still shows two transects that Alice is working on that need to go through NASCR.

Alice, let us know when things are complete and files are on AST4 so we can go ahead and close out this issue.

Thanks!

[alicebeittel]

Awesome! There are two transects that I need to run through the EV export script again after discovering issues Friday afternoon. My plan is to do them today.

LTJG Alice Beittel, NOAA Corps (she/her) Advanced Survey Technology Officer NOAA Southwest Fisheries Science Center 8901 La Jolla Shores Drive San Diego, CA 92037 Google Voice: (858) 546-5671 Cell: (510) 847-2048

On Mon, Dec 4, 2023 at 12:31 PM Josiah Renfree @.***> wrote:

Finished with my small subsections of SD-1096 transects, however the spreadsheet still shows two transects that Alice is working on that need to go through NASCR.

Alice, let us know when things are complete and files are on AST4 so we can go ahead and close out this issue.

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/52#issuecomment-1839428054, or unsubscribe https://github.com/notifications/unsubscribe-auth/A236EGUYTJQEN6R5B2XYHPLYHYXKZAVCNFSM6AAAAAA5KBSTOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZZGQZDQMBVGQ . You are receiving this because you were mentioned.Message ID: @.***>

[alicebeittel]

Finished my sections of Saildrone processing and the issues mentioned earlier. All files are uploaded to AST 4.

[kstierhoff]

I'm running plotCSV now with these new data. Did you still need me to plotSaildrone, or is that a moot point now that all the Saildrone data have been processed? Sorry that I missed that earlier request.

[alicebeittel]

Nope! No need to run plot Saildrone now. Let me know if there are any issues in the plotCSV that need to be addressed.

[kstierhoff]

Ok, I placed the updated plotCSV here: https://drive.google.com/drive/folders/1ZOv3hFg4S3TPsQsrCtt_yQeayinnRJCC?usp=drive_link

Most looks quite good. I'd check the following, not sure if errors or not:

• [x] Data from transect 096 is aligned with planned transect 097, and is short (without knowing where the SD actually traveled)
• [x] Data from 094 starts about half-way along the transects, perhaps what they actually surveyed
• [x] There's a gap in the offshore end of 134. Perhaps Ok.

[jrenfree]

Data from transect 096 is aligned with planned transect 097, and is short (without knowing where the SD actually traveled)

Yeah I think Saildrone just did the wrong line, at least in the old plotSaildrone you created:

Data from 094 starts about half-way along the transects, perhaps what they actually surveyed

They recorded FM pings for a large part of the eastern end of that line, so that data had to be discarded.

[alicebeittel]

134 is shorter because we shortened all of the Saildrone lines roughly 35nmi.

[kstierhoff]

Good copy on both. I'll probably exclude 096 since it overlaps with Lasker. The others look good as-is then.

Does this conclude the Saildrone data processing?

[alicebeittel]

Not yet.... I looked at 096 in EV with the GPS files and it looks like there is an issue with the gps file for Aug 13th (image, shows straight line over where 096 should be geographically). Opening the gps file for the 13th, it looks like there was an error in formatting the position for that day. I think i can change the decimal point and fix it. Then I'll see if we can process 096.

[jrenfree]

OK it's fixed! The updated, fixed version of 2023-08-13.gps.csv is now on AST4, and it indeed looks like they did survey Transect 096.

[alicebeittel]

Awesome! I'll take a look now

LTJG Alice Beittel, NOAA Corps (she/her) Advanced Survey Technology Officer NOAA Southwest Fisheries Science Center 8901 La Jolla Shores Drive San Diego, CA 92037 Google Voice: (858) 546-5671 Cell: (510) 847-2048

On Tue, Dec 5, 2023 at 2:23 PM Josiah Renfree @.***> wrote:

OK it's fixed! The updated, fixed version of 2023-08-13.gps.csv is now on AST4

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/52#issuecomment-1841720067, or unsubscribe https://github.com/notifications/unsubscribe-auth/A236EGRIY225WPZB47IZLZTYH6NGHAVCNFSM6AAAAAA5KBSTOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBRG4ZDAMBWG4 . You are receiving this because you were mentioned.Message ID: @.***>

[alicebeittel]

Okay! I replaced the 096 file with what should be the correct one in the right location. This should remove the displaced line and show new nasc bubbles on 096.

[kstierhoff]

Ok, I reran plotCSV-Saildrone, and posted the HTML file here: https://drive.google.com/drive/folders/1ZOv3hFg4S3TPsQsrCtt_yQeayinnRJCC?usp=drive_link

Looks like that fixed it.

[alicebeittel]

Great! Thanks Kevin. I know saildrone asked for our feedback on the data quality after we had a chance to look through everything. I think the only major issue we've come across is what Josiah discovered earlier on line

94. Half of it was FM.

Compared to last year it looked like the lines were relatively straighter. There were some deviations due to stopping mid-line and re-joining at the end of the day. The Saildrone were not always exactly back where they left off, but most of the time they were close enough. Josiah, did you find anything else in your lines?

LTJG Alice Beittel, NOAA Corps (she/her) Advanced Survey Technology Officer NOAA Southwest Fisheries Science Center 8901 La Jolla Shores Drive San Diego, CA 92037 Google Voice: (858) 546-5671 Cell: (510) 847-2048

On Tue, Dec 5, 2023 at 3:53 PM Kevin Stierhoff @.***> wrote:

Ok, I reran plotCSV-Saildrone, and posted the HTML file here:

https://drive.google.com/drive/folders/1ZOv3hFg4S3TPsQsrCtt_yQeayinnRJCC?usp=drive_link

Looks like that fixed it.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/52#issuecomment-1841822159, or unsubscribe https://github.com/notifications/unsubscribe-auth/A236EGT57BNJOZANVIJI3EDYH6XZDAVCNFSM6AAAAAA5KBSTOCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBRHAZDEMJVHE . You are receiving this because you were mentioned.Message ID: @.***>