Process Long Beach Carnage Data

[jrenfree]

11/10/2022: Scott has processed some of the data, which is now being processed through the R cps_nasc script by Alice. However there are still numerous transects that need to be processed in Echoview. Currently copying all LBC data to NAS2, which will be connected to the remote processing machine for someone to process remotely.

Current To-Do list:

• [x] Write script (Matlab or R) to generate new ECS files that are based off the template LBC ECS file but use the environment data from the transect-specific ECS files (which were taken from the Lasker)
• [x] Re-integrate all EV files using new, corrected ECS files
• [x] Apply CPS NASC mask to new CSV files
  • Files were re-processed but there were five transects that didn't have corresponding CPS NASC masks. Will check to see if they exist somewhere else, otherwise will have to run them manually through NASC(R) script.

[jrenfree]

@alicebeittel @ScottMau What is the status of the LBC processing? On AST3 I believe it shows that all of the nearshore transects have been processed, but perhaps the island transects have not. Is this accurate?

[ScottMau]

Josiah, I am working on the island transects now on the computer Murfin has set up. I should have ana capa done today and Catalina done by the end of the week. They are all stored on the portable drive attached to the processing computer since it can't be attached to the network.

Best regards,

Scott

On Wednesday, November 16, 2022, Josiah Renfree @.***> wrote:

@alicebeittel https://github.com/alicebeittel @ScottMau https://github.com/ScottMau What is the status of the LBC processing? On AST3 I believe it shows that all of the nearshore transects have been processed, but perhaps the island transects have not. Is this accurate?

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1317727044, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMQKW4527RUKPQKXEPDWIVKPXANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[jrenfree]

Great, thanks for the info. I'll hold off on doing anything with LBC data then until you are done.

Let me know if you want to split up any of the data processing, although not sure how easy that is when the processing is all done on that single computer.

[alicebeittel]

I have processed (extract_cps_nasc.r) all of the LBC data that Scott has uploaded to the server, which I believe is all of the data except for transects around the islands. Fixed the issue with 2207LBC_153N_CPS, re-exported the csv and reprocessed through nasc.r. Is there a way I can get access to the new csv files and echograms as Scott processes them? Or at least upload them in batches, so I can crack away on them this week? @ScottMau @dmurfin

[dmurfin]

Hi Alice. I believe all of the processed files and remaining raw files to be processed are on an external hard drive attached to the EV processing machine. You should be able to view and access all of this via the teamviewer group account that I (Renfree’s acct) sent to you this morning. Please be cognizant to communicate well with Scott so we do not duplicate efforts and/or data sets. Dave M.

On Nov 16, 2022, at 5:47 PM, Alice Beittel @.***> wrote:

 I have processed all of the LBC data that Scott has uploaded to the server, which I believe is all of the data except for transects around the islands. Fixed the issue with 2207LBC_153N_CPS, re-exported the csv and reprocessed through nasc.r. Is there a way I can get access to the new csv files and echograms as Scott processes them? Or at least upload them in batches, so I can crack away on them this week? @ScottMau @dmurfin

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[alicebeittel]

Thanks @dmurfin. Realistically I won't have time to get started until Monday due to the EV course which sounds like it will work well to transfer data over on Monday after Scott finishes this week.

[ScottMau]

Processing is done. Alice is working on the nasc.r today and should be able to get it finished. The remote processing went mostly well. It was quick and easy most of the time.

1. It would loose the keyboard occasionally so I had to shut down the entire computer for it to recognize it again.
2. I had it loose connection a few times but when it did the Echoveiw computer would be frozen where I left it so it was easy to continue once connection was reestablished.
3. I would prefer to do this with a bigger screen. Having the teamviewer screen makes things smaller so having a very large screen would be advantages.
4. Over all it worked but I don't see how it makes it any more convenient or expedient do to the extra steps involved with logging in.
5. one advantage is that it allows others access to the dongle and data while you are doing other things.
6. It unfortunately doesn't seam to allow multiple people to use the dongle at the same time do to the fact that you would be fighting over the keyboard and mouse to accomplish your tasks. Maybe there is another software that would allow this.

[dmurfin]

Excellent Scott. Thank you for the functional feedback. The connectivity issues are not something I readily know a solution to. The screen resolution on the control side (Your local Team view loggin) can be adjusted in many different ways to improve visibility. play around with the screen settings and optimize it for your monitor(s). Dave

On Mon, Nov 21, 2022 at 10:45 AM ScottMau @.***> wrote:

Processing is done. Alice is working on the nasc.r today and should be able to get it finished. The remote processing went mostly well. It was quick and easy most of the time.

1. It would loose the keyboard occasionally so I had to shut down the entire computer for it to recognize it again.
2. I had it loose connection a few times but when it did the Echoveiw computer would be frozen where I left it so it was easy to continue once connection was reestablished.
3. I would prefer to do this with a bigger screen. Having the teamviewer screen makes things smaller so having a very large screen would be advantages.
4. Over all it worked but I don't see how it makes it any more convenient or expedient do to the extra steps involved with logging in.
5. one advantage is that it allows others access to the dongle and data while you are doing other things.
6. It unfortunately doesn't seam to allow multiple people to use the dongle at the same time do to the fact that you would be fighting over the keyboard and mouse to accomplish your tasks. Maybe there is another software that would allow this.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1322497943, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4DZONPFSVTSEHKGRVA557LWJO7K3ANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- David Murfin http://swfsc.noaa.gov/staff.aspx?&id=632 Engineer NOAA Fisheries Service - Southwest Fisheries Science Center 8901 La Jolla Shores Dr. La Jolla, CA 92037 TE: (858) 546-7105 FX: (858) 546-5656

[alicebeittel]

Done processing all LBC files. Only remaining action item is to come to a consensus on transects 141N, 140N, and 139N where there are some curious returns in the shallow areas. We talked about them being kelp-like on Friday but could also be schools. See additional images below. @ddemer any thoughts on these? Top image is from 141N, bottom image is from 140N.

[alicebeittel]

I have only transferred over the new LBC csv files and exported images from the remote ev processing computer to AST3. Everything else has not been transferred yet.

[jrenfree]

So what exactly has not been transferred to AST3 yet? Just the few transects that you have some questions on?

[jrenfree]

Done processing all LBC files. Only remaining action item is to come to a consensus on transects 141N, 140N, and 139N where there are some curious returns in the shallow areas. We talked about them being kelp-like on Friday but could also be schools. See additional images below. @ddemer any thoughts on these? Top image is from 141N, bottom image is from 140N.

It's somewhat hard to tell from the frequency spectra, but it almost looks like that target has a resonance frequency of ~100 kHz, which would equate to something ~2-cm long, so a relatively small target. And the Sv is fairly high, so it could either be a dense school of a small weak scatterer (e.g., krill/zooplankton/copepods), or a single small but strong scatterer (e.g., kelp?).

Not entirely sure what we'd expect kelp to look like. But if it were rockfish or CPS I would expect the frequency spectra to be flatter and/or peak at a lower frequency (i.e., be higher at 38 or 70 kHz).

[alicebeittel]

Thanks for the input! I only copied over the CSVs and exported images for the island transects Scott finished today to AST3 for nasc.r. I did not move over any of the EV files, or ecs, or line files. I believe Scott has already transferred over all processed data for the coastal lines to AST 3 and is going bring over the rest this week. Right now the three lines in question are processed with all of the “kelp” looking returns included. Hope this clarifies things.

On Mon, Nov 21, 2022 at 16:22 Josiah Renfree @.***> wrote:

So what exactly has not been transferred to AST3 yet? Just the few transects that you have some questions on?

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1322830178, or unsubscribe https://github.com/notifications/unsubscribe-auth/A236EGSOWM2LUY6V5WCVO3TWJQG2FANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- LTJG Alice Beittel, NOAA Corps Advanced Survey Technology Officer NOAA Southwest Fisheries Science Center 8901 La Jolla Shores Drive San Diego, CA 92037 Google Voice: (541) 283-3756 Cell: (510) 847-2048

[jrenfree]

So @alicebeittel and @ScottMau can you confirm that all the LBC files are on AST3 and no longer exist anywhere else? I'm going to transfer the data over to the remote processing PC to do a little work, and want to make sure that (1) I'm using the correct data and (2) there aren't multiple copies floating around out there.

Thanks!

[ScottMau]

nope, Just transferred them now. I have them on the 4 tb drive in my office attached to my computer as well. You should be all set now.

Scott

On Tue, Nov 22, 2022 at 9:40 PM Josiah Renfree @.***> wrote:

So @alicebeittel https://github.com/alicebeittel and @ScottMau https://github.com/ScottMau can you confirm that all the LBC files are on AST3 and no longer exist anywhere else? I'm going to transfer the data over to the remote processing PC to do a little work, and want to make sure that (1) I'm using the correct data and (2) there aren't multiple copies floating around out there.

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1324576613, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMXYA24PGFRCNLYT3YLWJWU4FANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[ScottMau]

I will put the ast4 drive back on the Echoview processing computer so you have the data there as well.

Scott

On Wed, Nov 23, 2022 at 9:34 AM Scott Mau - NOAA Federal @.***> wrote:

nope, Just transferred them now. I have them on the 4 tb drive in my office attached to my computer as well. You should be all set now.

Scott

On Tue, Nov 22, 2022 at 9:40 PM Josiah Renfree @.***> wrote:

So @alicebeittel https://github.com/alicebeittel and @ScottMau https://github.com/ScottMau can you confirm that all the LBC files are on AST3 and no longer exist anywhere else? I'm going to transfer the data over to the remote processing PC to do a little work, and want to make sure that (1) I'm using the correct data and (2) there aren't multiple copies floating around out there.

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1324576613, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMXYA24PGFRCNLYT3YLWJWU4FANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[jrenfree]

Oh awesome that would be great, thanks!

On Wed, Nov 23, 2022 at 9:42 AM ScottMau @.***> wrote:

I will put the ast4 drive back on the Echoview processing computer so you have the data there as well.

Scott

On Wed, Nov 23, 2022 at 9:34 AM Scott Mau - NOAA Federal @.***> wrote:

nope, Just transferred them now. I have them on the 4 tb drive in my office attached to my computer as well. You should be all set now.

Scott

On Tue, Nov 22, 2022 at 9:40 PM Josiah Renfree @.***> wrote:

So @alicebeittel https://github.com/alicebeittel and @ScottMau https://github.com/ScottMau can you confirm that all the LBC files are on AST3 and no longer exist anywhere else? I'm going to transfer the data over to the remote processing PC to do a little work, and want to make sure that (1) I'm using the correct data and (2) there aren't multiple copies floating around out there.

Thanks!

— Reply to this email directly, view it on GitHub < https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1324576613 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ABSWGMXYA24PGFRCNLYT3YLWJWU4FANCNFSM6AAAAAASCWSH7Q

. You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0A+La+Jolla,+CA+92037-1508+%0D%0D%0A+Ph:+++858?entry=gmail&source=g

https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0A+La+Jolla,+CA+92037-1508+%0D%0D%0A+Ph:+++858?entry=gmail&source=g> La Jolla, CA 92037-1508 https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0A+La+Jolla,+CA+92037-1508+%0D%0D%0A+Ph:+++858?entry=gmail&source=g

https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0A+La+Jolla,+CA+92037-1508+%0D%0D%0A+Ph:+++858?entry=gmail&source=g> Ph: 858 https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0A+La+Jolla,+CA+92037-1508+%0D%0D%0A+Ph:+++858?entry=gmail&source=g -546-5645

Fax: 858-546-5651 Cell: 619-840-1394

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0ALa+Jolla,+CA+92037-1508+%0D%0D%0APh:+++858?entry=gmail&source=g La Jolla, CA 92037-1508 https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0ALa+Jolla,+CA+92037-1508+%0D%0D%0APh:+++858?entry=gmail&source=g Ph: 858 https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0ALa+Jolla,+CA+92037-1508+%0D%0D%0APh:+++858?entry=gmail&source=g -546-5645 Fax: 858-546-5651 Cell: 619-840-1394

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1325441351, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACD2JYGJWHV7Y6BCLMQBS43WJZJPPANCNFSM6AAAAAASCWSH7Q .

You are receiving this because you were assigned.Message ID: @.***>

-- Acoustic Engineer Advanced Survey Technologies Southwest Fisheries Science Center / NOAA 8901 La Jolla Shores Drive La Jolla, CA 92037 WORK: (858) 546-5669 CELL: (858) 232-3121

[jrenfree]

The CSV files on the remote processing PC do not match those on AST3, which is what I was worried about.

@ScottMau @alicebeittel Could you please figure out why two directories do not match? I need to process the CSV files on the remote processing machine, but I need to know if they are the current versions.

Also please let me know if the EV files do not also match, as if that's the case then I will need to reprocess the data again.

[alicebeittel]

Yikes. I can look into it right now. I'll check the EV files, CSV files, and the exported images.

LTJG Alice Beittel, NOAA Corps Advanced Survey Technology Officer NOAA Southwest Fisheries Science Center 8901 La Jolla Shores Drive San Diego, CA 92037 Google Voice: (541) 283-3756 Cell: (510) 847-2048

On Mon, Nov 28, 2022 at 9:38 AM Josiah Renfree @.***> wrote:

The CSV files on the remote processing PC do not match those on AST3, which is what I was worried about.

@ScottMau https://github.com/ScottMau @alicebeittel https://github.com/alicebeittel Could you please figure out why two directories do not match? I need to process the CSV files on the remote processing machine, but I need to know if they are the current versions.

Also please let me know if the EV files do not also match, as if that's the case then I will need to reprocess the data again.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1329484192, or unsubscribe https://github.com/notifications/unsubscribe-auth/A236EGX7NY3TF42JFKIPDW3WKTUX7ANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

[jrenfree]

Thanks.

I have a directory in there called “REPROCESSING.” Please don’t change anything in that directory.

On Mon, Nov 28, 2022 at 9:50 AM Alice Beittel @.***> wrote:

Yikes. I can look into it right now. I'll check the EV files, CSV files, and the exported images.

LTJG Alice Beittel, NOAA Corps Advanced Survey Technology Officer NOAA Southwest Fisheries Science Center 8901 La Jolla Shores Drive https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0ASan+Diego,+CA+92037?entry=gmail&source=g San Diego, CA 92037 https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0ASan+Diego,+CA+92037?entry=gmail&source=g Google Voice: (541) 283-3756 Cell: (510) 847-2048

On Mon, Nov 28, 2022 at 9:38 AM Josiah Renfree @.***> wrote:

The CSV files on the remote processing PC do not match those on AST3, which is what I was worried about.

@ScottMau https://github.com/ScottMau @alicebeittel https://github.com/alicebeittel Could you please figure out why two directories do not match? I need to process the CSV files on the remote processing machine, but I need to know if they are the current versions.

Also please let me know if the EV files do not also match, as if that's the case then I will need to reprocess the data again.

— Reply to this email directly, view it on GitHub < https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1329484192 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/A236EGX7NY3TF42JFKIPDW3WKTUX7ANCNFSM6AAAAAASCWSH7Q

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1329508690, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACD2JYCRMUB7ZJFLK5XYX6DWKTWG5ANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were assigned.Message ID: @.***>

-- Acoustic Engineer Advanced Survey Technologies Southwest Fisheries Science Center / NOAA 8901 La Jolla Shores Drive La Jolla, CA 92037 WORK: (858) 546-5669 CELL: (858) 232-3121

[alicebeittel]

Okay, I think I figured out the issue. On the remote PC, it looks like files that were processed prior to 11/10/2022 were saved on the ast-data drive (169.254.215.20) and files processed after that date were saved to the AST4TB external drive. Files from the ast-data drive prior to 11/10 also seem to have been copied over to AST4TB (I checked to make sure everything matches). There are 6 transects that were processed through nasc on ast-data drive (184-189) that did not make it to the AST4TB drive. Those 6 nasc files can be deleted since they were not in the files I copied over and I processed them separately and directly to AST3 server. Everything on the AST3 server matches the AST4TB external drive but will not match ast-data drive since it looks like ast-data has only files processed prior to 11/10.

Should the Carnage Processed EV folder be saved to the ast-data drive or to the AST4TB external hard drive? I can organize things accordingly. Hope this helps clear things up.

On Mon, Nov 28, 2022 at 9:53 AM Josiah Renfree @.***> wrote:

Thanks.

I have a directory in there called “REPROCESSING.” Please don’t change anything in that directory.

On Mon, Nov 28, 2022 at 9:50 AM Alice Beittel @.***> wrote:

Yikes. I can look into it right now. I'll check the EV files, CSV files, and the exported images.

LTJG Alice Beittel, NOAA Corps Advanced Survey Technology Officer NOAA Southwest Fisheries Science Center 8901 La Jolla Shores Drive < https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0ASan+Diego,+CA+92037?entry=gmail&source=g

San Diego, CA 92037 < https://www.google.com/maps/search/8901+La+Jolla+Shores+Drive+%0D%0D%0ASan+Diego,+CA+92037?entry=gmail&source=g

Google Voice: (541) 283-3756 Cell: (510) 847-2048

On Mon, Nov 28, 2022 at 9:38 AM Josiah Renfree @.***> wrote:

The CSV files on the remote processing PC do not match those on AST3, which is what I was worried about.

@ScottMau https://github.com/ScottMau @alicebeittel https://github.com/alicebeittel Could you please figure out why two directories do not match? I need to process the CSV files on the remote processing machine, but I need to know if they are the current versions.

Also please let me know if the EV files do not also match, as if that's the case then I will need to reprocess the data again.

— Reply to this email directly, view it on GitHub <

https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1329484192

, or unsubscribe <

https://github.com/notifications/unsubscribe-auth/A236EGX7NY3TF42JFKIPDW3WKTUX7ANCNFSM6AAAAAASCWSH7Q

. You are receiving this because you were mentioned.Message ID: @.***>

— Reply to this email directly, view it on GitHub < https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1329508690 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/ACD2JYCRMUB7ZJFLK5XYX6DWKTWG5ANCNFSM6AAAAAASCWSH7Q

. You are receiving this because you were assigned.Message ID: @.***>

-- Acoustic Engineer Advanced Survey Technologies Southwest Fisheries Science Center / NOAA 8901 La Jolla Shores Drive La Jolla, CA 92037 WORK: (858) 546-5669 CELL: (858) 232-3121

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1329511432, or unsubscribe https://github.com/notifications/unsubscribe-auth/A236EGX3YIKIUD5AWBSHGG3WKTWPXANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

[jrenfree]

Just to ensure we're talking about the same thing, is the following accurate:

AST4TB: 4 TB external drive currently connected to the remote processing PC (G:\ drive) ast-data: Synology NAS drive in Murfin's office connected to remote processing PC AST3: Server on lab's network with our main survey directory

What confuses me is that you say everything on AST3 matches that on AST4TB, however I ran a SyncBack simulation earlier this morning and saw differences in the CSV folder.

Ideally nothing should be on the ast-data drive, but we need to ensure that there are no files on there that need to be backed up to AST3.

[ScottMau]

All, I did everything for LBC on the AST4TB drive except for most of the nasc.r files that Alice did. I assume by your scan that she put the NASC.R files in the CSV file on AST3 which would give you the differences you see when running the syncback simulation. Alice am I correct with this assumption?

Scott

On Mon, Nov 28, 2022 at 11:31 AM Josiah Renfree @.***> wrote:

Just to ensure we're talking about the same thing, is the following accurate:

AST4TB: 4 TB external drive currently connected to the remote processing PC (G:\ drive) ast-data: Synology NAS drive in Murfin's office connected to remote processing PC AST3: Server on lab's network with our main survey directory

What confuses me is that you say everything on AST3 matches that on AST4TB, however I ran a SyncBack simulation earlier this morning and saw differences in the CSV folder.

Ideally nothing should be on the ast-data drive, but we need to ensure that there are no files on there that need to be backed up to AST3.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1329638074, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMU6KWX5XHYQQW33AW3WKUCBVANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[alicebeittel]

Yes, all of those definitions are correct/what I was referring too. Thanks for checking.

Ok, I see what you mean now. I checked the # of files but didn't think about running a simulated SyncBack to see differences in date. I just rant it between AST4TB and AST3. For the csv conflicts: 153N_CPS-Final 38 kHz CPS.csv - I had to reprocess and export the csv, I did this directly to AST3 so that's why there is an older version on AST4TB. Keep version on AST3. 158N_CPS-Final 38 kHz CPS.csv - not sure why there is a newer version of the csv on AST4TB unless it was reprocessed by scott after I moved it over to AST3. 195N_CPS-Final 38 kHz_CPS_nasc_cps (.csv and .png) - these must have been processed by scott and then re-processed by me on AST3. For consistency, I would say we keep the ones on AST3. 211N_CPS-Final 38 kHz_CPS_nasc_cps (.csv and .png) - these must have been processed by scott and then re-processed by me on AST3. For consistency, I would say we keep the ones on AST3.

For the EV conflicts: 140N_CPS - I opened and re-saved the EV file for backscatter deciphering for group review 141N_CPS - Not sure about this one, but likely I opened it to look at something and re-saved. 153N_CPS - issue with seabed - I reprocessed to AST3. Keep version on AST3. 184N_CPS - I opened and re-saved the EV file for backscatter deciphering for group review

Does this cover all the conflicts you saw on your end too?

I also see some conflicts in the Exported Images (all three .png files for 371N) but am not sure why those don't match when the csvs match. The only conflict I see for Exported Lines is for 153N which makes sense since I needed to re-do the integration stop line.

[jrenfree]

I didn't make a list of which transects had conflicts, but that number seems about right.

Can you go ahead and copy over the necessary files so that the versions you believe are correct are the same on both AST4TB and AST3?

After that's done, can you confirm which EV files on AST4TB might be substantially different than what was previously on AST4TB? It sounds like it might just be 153N.

I'm not too worried about the images, so you can handle those as you see fit.

Thanks.

[alicebeittel]

Done! I copied over 153 EV file and line file from AST3 to AST4TB (also changed the raw file and ecs links so it will work with the remote PC file path names). The other EV files I copied over the versions from AST4TB to AST3 so that they would be the same and preserve the remote PC file path names.

For the nasc files and exported images, I copied over the versions from AST3 to AST4TB.

Everything should match now!

[jrenfree]

@ScottMau I looked a closer look at one of the LBC EV files, and it appears that the ECS file it uses is directly from the Lasker processing.

Did you create Carnage-specific ECS files using the Carnage ECS template with Lasker CTD info, or did you just use an ECS file from the Lasker directory? It seems like it might be the latter, in which case we'll need to generate new ECS files and re-process the data. But I want to confirm with you before going down that route.

Thanks.

[ScottMau]

I did not have a Carnage ecs template. We never had an ECS template in the past. Just a Carnage specific ev template. If you think we need it I can start making new ecs files tomorrow. I don't even know how it would be different. Doesn't the ECS file just change the parameters of the ev file based on the water characteristics of the uctd data? let me know how you want to proceed.

Scott

On Mon, Nov 28, 2022 at 8:03 PM Josiah Renfree @.***> wrote:

@ScottMau https://github.com/ScottMau I looked a closer look at one of the LBC EV files, and it appears that the ECS file it uses is directly from the Lasker processing.

Did you create Carnage-specific ECS files using the Carnage ECS template with Lasker CTD info, or did you just use an ECS file from the Lasker directory? It seems like it might be the latter, in which case we'll need to generate new ECS files and re-process the data. But I want to confirm with you before going down that route.

Thanks.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1330043935, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMTGQBBUKZKIPYLD7F3WKV6ALANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[jrenfree]

In addition to specifying the water characteristics, the ECS files also contain calibration parameters for each of the transducers. Which is what tipped me off to this issue in the first place, in that Echoview was telling me it couldn't find transducer T5 (333 kHz) that was specified in the ECS file (because it came from the Lasker).

Thankfully the fix can be done via scripting, so I'll get working on that. For now don't believe I'll need any assistance.

[ScottMau]

Ok, Let me know if you need me for anything else. I will be at the lab today working on my dive gear. best regards,

Scott

On Tue, Nov 29, 2022 at 8:02 AM Josiah Renfree @.***> wrote:

In addition to specifying the water characteristics, the ECS files also contain calibration parameters for each of the transducers. Which is what tipped me off to this issue in the first place, in that Echoview was telling me it couldn't find transducer T5 (333 kHz) that was specified in the ECS file (because it came from the Lasker).

Thankfully the fix can be done via scripting, so I'll get working on that. For now don't believe I'll need any assistance.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1330874261, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMRA37UWNMEQXJ3WMYDWKYSHDANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[jrenfree]

@alicebeittel It doesn't appear that there are regular CSV nor nasc_cps.csv files for the following transects:

• [x] 212
• [x] 213
• [x] 214
• [x] 215
• [x] 223
• [x] 224

Do you know if these exist somewhere? I've created new regular CSV files, but need to know if we have to manually run them through NASC(R) or if we have existing nasc_cps.csv files I can use.

Thanks!

[ScottMau]

I think that is the way they ran the lines that these were all in a row. I looked at the csv file and you are correct the lines were not run through the exporting script. If I didn't create a csv file then Alice didn't run it through the narc.r code either. I can run them through tomorrow if you like.

Scott

Scott

On Tue, Nov 29, 2022 at 5:14 PM Josiah Renfree @.***> wrote:

@alicebeittel https://github.com/alicebeittel It doesn't appear that there are regular CSV nor nasc_cps.csv files for the following transects:

• 212
• 213
• 214
• 215
• 223
• 224

Do you know if these exist somewhere? I've created new regular CSV files, but need to know if we have to manually run them through NASC(R) or if we have existing nasc_cps.csv files I can use.

Thanks!

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1331519979, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMX4UPUQZFXJO432H4LWK2TADANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[jrenfree]

Hi Scott (or Alice),

I copied the new CSV files over to AST3, in case either of you are willing and able to run them through the R script. They are located at:

20220627_CARNAGE_SummerCPS\PROCESSED\EV\CSV\TO_FIX

I can do them as well, but I might be working from home tomorrow so might be easier for one of you to do it at the lab.

Thanks.

[ScottMau]

I found the original CSV files so I don't know what happened to them after you re ran them with the script for the new ecs files. I will get on these asap. Where do you want me to upload them?

Scott

On Tue, Nov 29, 2022 at 10:14 PM Josiah Renfree @.***> wrote:

Hi Scott (or Alice),

I copied the new CSV files over to AST3, in case either of you are willing and able to run them through the R script. They are located at:

20220627_CARNAGE_SummerCPS\PROCESSED\EV\CSV\TO_FIX

I can do them as well, but I might be working from home tomorrow so might be easier for one of you to do it at the lab.

Thanks.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1331691087, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMTPVW5VGGXWE2WWPVTWK3WCRANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[jrenfree]

Finished processing all of the LBC data, and copied the "PROCESSED" folder over to AST3, but will wait to close out this Issue until I can physically compare the external HDD to AST3.

[jrenfree]

All data transferred back to AST3 drive and deleted from external HDD.

[kstierhoff]

Hoping @ScottMau or @alicebeittel can help me out.

LBC transect 157 must be reprocessed in Echoview and R; it looks to contain the transit to Santa Cruz, instead of the transect data.

LBC transects 203 and 204 are missing, or not sampled, or I can't find them. Any intel on these two lines?

Thanks, Kevin

[alicebeittel]

I can take a look at these this afternoon.

[ScottMau]

Lines 203 and 204 did not have any raw files associated with them. I will process 157 right now.

scott

On Wed, Dec 7, 2022 at 1:07 PM Alice Beittel @.***> wrote:

I can take a look at these this afternoon.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1341598194, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMVIPJWYI6VYBXCMUQDWMD4ALANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[alicebeittel]

@ScottMau Not sure if you saw my earlier reply in the thread. I already opened the 157 EV file on the server and started digging into it. Should I stop?

@jrenfree if there are changes to the EV file on the remote PC, we will need to make sure that change is copied over to the server, correct? I can do this if so.

[jrenfree]

I don't believe the remote PC shouldn't have any LBC data on it. Is that wrong?

[ScottMau]

No that is fine. I have other stuff to work on. Plus I don't know what happened to the file on the remote pc because it doesn't appear to have a transect line at all. It has the ship going to Santa cruz but nothing where the line is suppose to be.

scott

On Wed, Dec 7, 2022 at 1:46 PM Alice Beittel @.***> wrote:

@ScottMau https://github.com/ScottMau Not sure if you saw my earlier reply in the thread. I already opened the 157 EV file on the server and started digging into it. Should I stop?

@jrenfree https://github.com/jrenfree if there are changes to the EV file on the remote PC, we will need to make sure that change is copied over to the server, correct? I can do this if so.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1341636320, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMQHGS2LI245MRPJ443WMEAS7ANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[ScottMau]

It does have the LBC data on it. It is on the W: synology drive.

Scott

On Wed, Dec 7, 2022 at 1:49 PM Josiah Renfree @.***> wrote:

I don't believe the remote PC shouldn't have any LBC data on it. Is that wrong?

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1341639272, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMRBLN32URS5GK2E6LTWMEA75ANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[jrenfree]

The LBC processing was last being done an external 4 TB HDD. I would not work on anything for LBC that's located on the Synology drive.

[ScottMau]

I believe Alice is using the data from the server for line 157. Do you have the 4 TB drive? I don't see it on the processing computer.

Scott

On Wed, Dec 7, 2022 at 1:57 PM Josiah Renfree @.***> wrote:

The LBC processing was last being done an external 4 TB HDD. I would not work on anything for LBC that's located on the Synology drive.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1341646178, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMSRKYLN5VBRFOU4AZ3WMEB5VANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[jrenfree]

I removed it from the processing PC, backed everything up to AST3, then deleted the data. I don't think I was aware there was still data on Synology.

The most up-to-date data should be AST3, so whatever needs to be done should be obtained from there.

[ScottMau]

Josiah, Thank you for the clarification. I will let Alice reprocess 157 on her machine with the data from AST3.

Scott

On Wed, Dec 7, 2022 at 2:09 PM Josiah Renfree @.***> wrote:

I removed it from the processing PC, backed everything up to AST3, then deleted the data. I don't think I was aware there was still data on Synology.

The most up-to-date data should be AST3, so whatever needs to be done should be obtained from there.

— Reply to this email directly, view it on GitHub https://github.com/nmfs-swfsc-ast/ast-tasks/issues/14#issuecomment-1341657056, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABSWGMXVIFQLAJBIGOKDWJ3WMEDH7ANCNFSM6AAAAAASCWSH7Q . You are receiving this because you were mentioned.Message ID: @.***>

-- Scott A. Mau Research Fisheries Biologist / Unit Diving Supervisor Fisheries Resources Division Southwest Fisheries Science Center 8901 La Jolla Shores Drive La Jolla, CA 92037-1508 Ph: 858-546-5645 Fax: 858-546-5651 Cell: 619-840-1394

[alicebeittel]

@kstierhoff 157 reprocessed in EV, exported, and reprocessed in nasc.r from AST3 server.

203 and 204 are not in the same day/time frame as the other lines in the area. There is no data where the black line cuts through land (it looks like EV just joins the end of one line to the start of the next no matter where they are). It could be that they came back at a later time/date to these lines, I'll check the survey notes.

[alicebeittel]

The survey notes have start and stop times for 203 and 204 for 02 Aug 2022 (roughly 30 min of survey time for each line), but there is no data that matches that time/date on AST3. @jrenfree Do we still have the original hard drive where the data was recorded to on LBC? Trying to cover all the bases here before saying we don't have any data for those lines. The lines should be later in the day just after file D20220802-T185625. Happy to dig around and look for them on another drive if it exists.

[jrenfree]

I see file "2207LBC-D20220802-T193811.raw" on AST3. Is it even later than that?