search

USF-IMARS / erddap-config

Content dir for docker-erddap incl setup.xml & dataset.xml
0 stars 1 forks source link

get ERDDAP back up #30

Closed 7yl4r closed 1 year ago

7yl4r commented 1 year ago
7yl4r commented 1 year ago

related: https://github.com/USF-IMARS/imars_puppet_v3/issues/3

started the rsync:

[root@dune puppetlabs]# rsync -hazv /srv/imars-objects/fk/MEAN_7D_MODA/OC/* /mnt/sdb/moda_oc_7d_fk/. | tee ~/moda_oc_7d_fk-manual_rsync-2022_11_18.log
sending incremental file list
MODA_2002185_2002189_7D_FK_OC.nc
MODA_2002190_2002196_7D_FK_OC.nc
MODA_2002197_2002203_7D_FK_OC.nc
MODA_2002204_2002210_7D_FK_OC.nc
MODA_2002219_2002224_7D_FK_OC.nc
MODA_2002225_2002231_7D_FK_OC.nc
MODA_2002233_2002238_7D_FK_OC.nc
[...]

I just restarted the docker container with the new config. It should be able to load the dataset even as files are still streaming in.

7yl4r commented 1 year ago

It works! :raised_hands:

@dotis : we need to choose one of the ways to work with this on the processing server you use:

  1. use NFS to access the data on the ERDDAP server directly
  2. copy the data from the processing server to the ERDDAP server whenver it needs updating

(1) is nice because no duplication of data means less disk usage & less confusion.

On the ohter hand the duplicate data in (2) means that reprocessing & testing can happen while leaving the previous version of the data up.

dotis commented 1 year ago

Excellent. I think for now, we should perhaps write the 7D mean files to two places. I can write them to the current directory and keep things consistent and I can also write a set to the ERDDAP server.

We could streamline later if needed. Do you have a path for the ERDDAP server?

Thanks,

From: Tylar @.> Sent: Tuesday, November 22, 2022 1:33 PM To: USF-IMARS/erddap-config @.> Cc: dotis @.>; Mention @.> Subject: Re: [USF-IMARS/erddap-config] get ERDDAP back up (Issue #30)

It works! 🙌

@dotishttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotis&data=05%7C01%7Cdotis%40usf.edu%7C18e63e8594d448716e2c08daccb80af6%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638047388089253078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BA7YWUQEdzcU8D5OdQuNKSPq8YygIVGqGM7XL%2Bao5Nk%3D&reserved=0 : we need to choose one of the ways to work with this on the processing server you use:

  1. use NFS to access the data on the ERDDAP server directly
  2. copy the data from the processing server to the ERDDAP server whenver it needs updating

(1) is nice because no duplication of data means less disk usage & less confusion.

On the ohter hand the duplicate data in (2) means that reprocessing & testing can happen while leaving the previous version of the data up.

— Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSF-IMARS%2Ferddap-config%2Fissues%2F30%23issuecomment-1324090521&data=05%7C01%7Cdotis%40usf.edu%7C18e63e8594d448716e2c08daccb80af6%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638047388089253078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qvroOGUDSx3MzBgpG5Ecsho2P39Y8ESel%2FYTCwkFrAU%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGKS3DKJE5DEA3B5EEAX4B3WJUGXJANCNFSM6AAAAAASE2CXZM&data=05%7C01%7Cdotis%40usf.edu%7C18e63e8594d448716e2c08daccb80af6%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638047388089253078%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=WC3oJNIf%2B51SkqDdn4kRFwQRCHMrPpR7q2VUD9M4Hnw%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.

7yl4r commented 1 year ago

This is the command to run on dune that copies your files to the ERDDAP location:

rsync -hazv /srv/imars-objects/fk/MEAN_7D_MODA/OC/* /mnt/sdb/moda_oc_7d_fk/.

To run this on manglillo (or seashell) I think you can do:

rsync -hazv /srv/imars-objects/fk/MEAN_7D_MODA/OC/* dotis@dune:/mnt/sdb/moda_oc_7d_fk/.

If that works without issue then we can schedule the rsync command in your crontab.

dotis commented 1 year ago

Ok. I am going to get the dashboard files up and current again this week and will try this command. I have had some inquiries from NOAA and their Iconic Reef Program. They seem like a good potential end-user. I may add some extraction points/boxes at the Iconic Reef sites. Or, I suppose we could even develop an "Iconic Reef" dash if there is enough interest. Will Google charges go up as we add more boards?

Thanks!

From: Tylar @.> Sent: Monday, November 28, 2022 2:05 PM To: USF-IMARS/erddap-config @.> Cc: dotis @.>; Mention @.> Subject: Re: [USF-IMARS/erddap-config] get ERDDAP back up (Issue #30)

This is the command to run on dune that copies your files to the ERDDAP location:

rsync -hazv /srv/imars-objects/fk/MEAN_7D_MODA/OC/* /mnt/sdb/moda_oc_7d_fk/.

To run this on manglillo (or seashell) I think you can do:

rsync -hazv /srv/imars-objects/fk/MEAN_7D_MODA/OC/* @.***:/mnt/sdb/moda_oc_7d_fk/.

If that works without issue then we can schedule the rsync command in your crontab.

— Reply to this email directly, view it on GitHubhttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FUSF-IMARS%2Ferddap-config%2Fissues%2F30%23issuecomment-1329604506&data=05%7C01%7Cdotis%40usf.edu%7C0186d7983dcc401c34a008dad17388ce%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638052591393417910%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=RgYGawCBCBBtQNaapevjttusD3ijwcsalnkW%2F8KZNgc%3D&reserved=0, or unsubscribehttps://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGKS3DK2LQIRMIICTOHPTF3WKT677ANCNFSM6AAAAAASE2CXZM&data=05%7C01%7Cdotis%40usf.edu%7C0186d7983dcc401c34a008dad17388ce%7C741bf7dee2e546df8d6782607df9deaa%7C0%7C0%7C638052591393574139%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iablb8ZFYnoUBAu6FBchLmM6pJoO30ki9Jk3ernAoSs%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the sender and know the content is safe.

7yl4r commented 1 year ago

Google charges go up as we add more servers. So far we have been doing one board per server, but we should consider putting multiple boards onto one server. I think we can use one SE-US server to host the dashboards for FKNMS, FWC, and this new project.

dotis commented 1 year ago

This is what happened when I tried the rsync command. It seems to want to make an fgb directory, which is weird.

These directories are now up to date and ready for testing: /srv/imars-objects/fk/MEAN_7D_MODA/OC /srv/imars-objects/fk/MEAN_7D_MODA/SST /srv/imars-objects/fk/MEAN_7D_MODA/SST4

seashell 260 :/srv/imars-objects/fgb/MEAN_7D_MODA/SST4> rsync -hazv \
  /srv/imars-objects/fk/MEAN_7D_MODA/OC/* \
  ***@***.***/mnt/sdb/moda_oc_7d_fk/.

sending incremental file list
rsync: mkdir ***@***.***/mnt/sdb/moda_oc_7d_fk/." failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]
7yl4r commented 1 year ago

It looks like some of this got censored and that is making it hard for me to tell what has happened. Please email me the output directly.