maxwnewcomer / NWTOPT

NWTOPT is a hyperparameter optimization and tuning system for steady-state and transient models ran by MODFLOW-NWT
Creative Commons Zero v1.0 Universal
2 stars 0 forks source link

timeout units #15

Closed mnfienen closed 3 years ago

mnfienen commented 3 years ago

Looks like the example run.sh script says to enter timeout value in minutes but the code is converting to seconds.....just need to choose :)

maxwnewcomer commented 3 years ago

Here's the code for extracting time from run.sh (in NWT_SUBMIT/NWTOPT_FILES/objective.py, runModel()). timelim is what is used to timeout the modflow process. timelim needs to be in seconds given the function that is using it (subprocess.run). As you can see in the the try: except: the last line of run.sh is converted to seconds by multiplying by 60 and is set. It is then divided by 60 inline so that we can display the time out in minutes to the user (I've been assuming that setting timeout in minutes is much more convenient than setting in seconds). If the float() conversion fails, then there must be an errored input in the last line of run.sh, or it must be empty, resulting in timelim being set to none and the program informing the user of this. Let me know if you still think this is flawed, but after some tests of my own, it seems to be working (I put in a timeout of 12.5 minutes for a run and the upper bound of time elapsed for all the runs was 750 seconds, which checks out)

with open(os.path.join(cwd, 'run.sh')) as f:
        for line in f:
            if use_next:
                run_command = line
                use_next = False
            elif line.startswith('# Run Command:'):
                use_next = True
            last_line = line
    try:
        timelim = float(last_line) * 60
        print(f'[INFO] Timeout for model run is set to {timelim / 60} minutes')
    except Exception as e:
        timelim = None
        use_timer = False
        print('[INFO] No timeout set for model run')
    print(f'[INFO] Using run command: {run_command.strip()}')
    print(f'[INFO] Starting run out of {cwd}')
mnfienen commented 3 years ago

oh - that makes sense. Something else must be going on, then. Both @hwreeves-USGS and I were trying runs and the runs were writing a little output to the screen and then quitting. It seemed the amount of output they were writing was consistent with the timeout length if it was in seconds. So....maybe barking up the wrong tree :).

I tried a run with a much longer timeout and now no model output is being printed to the logs at all, but I'm not sure I gracefully killed the previous run.

maxwnewcomer commented 3 years ago

hmmmm, do you have any of that output? are you running through condor? just a note, each machine should be printing out its output and all error messages it received while running, in that order. Also note, if a run is terminated by the timeout it should say "[WARNING] Time Limit reached, terminating run", and if it runs to completion it should say "[INFO] Successful termination of trial"

mnfienen commented 3 years ago

ha! actually, discovered that when I set the timeout in seconds, it was too big (saw that in the logfile). I just reset to something reasonable and off and running again

maxwnewcomer commented 3 years ago

ahhh, makes sense. let me know if it works smoothly this time

mnfienen commented 3 years ago

hmmmm.......it's running now and screen output is like


 Using NAME file: gv3.nam 
 Run start date and time (yyyy/mm/dd hh:mm:ss): 2021/09/21 20:28:14

 Solving:  Stress period:     1    Time step:     1    Groundwater-Flow Eqn.

[INFO] Successful termination of trial

this is a transient model, though, so maybe it's only working on the first stress period. Is there something special to inform the code that my model is transient?

maxwnewcomer commented 3 years ago

Also just for reference: This is what the errors log looks like after a successful (timeout or no timeout) run:

INFO:hyperopt.mongoexp:PROTOCOL mongo
INFO:hyperopt.mongoexp:USERNAME None
INFO:hyperopt.mongoexp:HOSTNAME 130.11.177.99
INFO:hyperopt.mongoexp:PORT 27017
INFO:hyperopt.mongoexp:PATH /db/jobs
INFO:hyperopt.mongoexp:AUTH DB None
INFO:hyperopt.mongoexp:DB db
INFO:hyperopt.mongoexp:COLLECTION jobs
INFO:numexpr.utils:Note: NumExpr detected 20 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:hyperopt.mongoexp:job finished: 614956d15ce150f0e5244685
maxwnewcomer commented 3 years ago

And this is what a successful run no timeout should look like in logs/outputs:

[INFO] Timeout for model run is set to 12.5 minutes
[INFO] Using run command: wine MODFLOW-NWT_64.exe tmr_pr_11_13_calib_PM_flooded_scen3.nam
[INFO] Starting run out of /var/lib/condor/execute/dir_4126325/NWT_SUBMIT/PROJECT_FILES

                                  MODFLOW-NWT-SWR1 
    U.S. GEOLOGICAL SURVEY MODULAR FINITE-DIFFERENCE GROUNDWATER-FLOW MODEL
                             WITH NEWTON FORMULATION
                             Version 1.1.4 4/01/2018                         
                    BASED ON MODFLOW-2005 Version 1.12.0 02/03/2017                       

                    SWR1 Version 1.04.0 09/15/2016                       

 Using NAME file: tmr_pr_11_13_calib_PM_flooded_scen3.nam 
 Run start date and time (yyyy/mm/dd hh:mm:ss): 2021/09/20 23:01:23

 Solving:  Stress period:     1    Time step:     1    Groundwater-Flow Eqn.
 Run end date and time (yyyy/mm/dd hh:mm:ss): 2021/09/20 23:08:07
 Elapsed run time:  6 Minutes, 43.742 Seconds

  Normal termination of simulation

[INFO] Successful termination of trial
[MASS BALANCE]: 0.01
[SECONDS]: 403.742
[TOTAL ITERATIONS]: 40.0
mnfienen commented 3 years ago

shoot - I had an error mkdir: cannot create directory 'NWT_SUBMIT/PROJECT_FILES/nwts': File exists I guess I need to fix that....

maxwnewcomer commented 3 years ago

that's totally normal, i'm assuming there is just a nwts directory in your project_files already. either way its a nwtopt.sh error and shouldn't impact your run

maxwnewcomer commented 3 years ago

nwtopt.sh is the script sent and ran by condor to start each worker

maxwnewcomer commented 3 years ago

Sorry I just saw that you edited your previous comment, did the transient model issue work itself out or no?

mnfienen commented 3 years ago

No - transient models are still stopping after a single stress period but also don't seem to complete the sequence (see below)

I'm getting this for error file which is consistent with what you posted

INFO:hyperopt.mongoexp:USERNAME None
INFO:hyperopt.mongoexp:HOSTNAME 130.11.177.99
INFO:hyperopt.mongoexp:PORT 2742
INFO:hyperopt.mongoexp:PATH /db/jobs
INFO:hyperopt.mongoexp:AUTH DB None
INFO:hyperopt.mongoexp:DB db
INFO:hyperopt.mongoexp:COLLECTION jobs
INFO:numexpr.utils:Note: NumExpr detected 40 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
INFO:hyperopt.mongoexp:job finished: 614a889bae0fcc8c8df31d5c```

but the output file doesn't seem to be completing and is somehow terminating after one stress period.

var [INFO] Timeout for model run is set to 350.0 minutes [INFO] Using run command: ./mfnwt gv3.nam [INFO] Starting run out of /var/lib/condor/execute/dir_40943/NWT_SUBMIT/PROJECT_FILES

                              MODFLOW-NWT-SWR1 
U.S. GEOLOGICAL SURVEY MODULAR FINITE-DIFFERENCE GROUNDWATER-FLOW MODEL
                         WITH NEWTON FORMULATION
                         Version 1.1.4 4/01/2018                         
                BASED ON MODFLOW-2005 Version 1.12.0 02/03/2017                       

                SWR1 Version 1.04.0 09/15/2016                       

Using NAME file: gv3.nam Run start date and time (yyyy/mm/dd hh:mm:ss): 2021/09/21 20:36:28

Solving: Stress period: 1 Time step: 1 Groundwater-Flow Eqn.

[INFO] Successful termination of trial

maxwnewcomer commented 3 years ago

Hmmmm, if you run the pull_nwts.py script and pull the results, look in the nwt_performance.csv file and see if all the runs are erroring out, timing out, or if they are reporting back normal time elapsed and mbs

maxwnewcomer commented 3 years ago

You can run pull_nwts concurrently with the normal process btw. Also if you want it to continuously pull, use argument —loop True

mnfienen commented 3 years ago

yeah - not looking good.....


NWT Number,Loss,Mass Balance,Seconds Elapased,# of Iterations,Min Loss
5,999999999999.0,999999,999999,-1,9999
7,999999999999.0,999999,999999,-1,9999
20,999999999999.0,999999,999999,-1,9999
21,999999999999.0,999999,999999,-1,9999
22,999999999999.0,999999,999999,-1,9999
23,999999999999.0,999999,999999,-1,9999
24,999999999999.0,999999,999999,-1,9999
25,999999999999.0,999999,999999,-1,9999
26,999999999999.0,999999,999999,-1,9999
maxwnewcomer commented 3 years ago

I would say wait for other trials to come in. Pls note that the initial runs are all random

mnfienen commented 3 years ago

right on - thanks for looking into all this! Imma let it run to completion and check it out later

maxwnewcomer commented 3 years ago

Of course! Let me know how it goes!

maxwnewcomer commented 3 years ago

This does bring up an issue of run documentation, I'll add something to print when the run successfully completes but errors out due to bad solver settings

maxwnewcomer commented 3 years ago

Make sure you use screen so the process doesn't terminate prematurely - even if it does you can resume by using the same key

hwreeves-USGS commented 3 years ago

To add some more information to what Mike sent last night - here is an outfile and errors file - MODFLOW seems to be stopping right after it starts, these runs should take a couple of hours. I asked for 300 trials and 300 cores - in 5 minutes or so 298 of the cores finished errors_1710_103.txt outfile_1710_103.txt .

hwreeves-USGS commented 3 years ago

I am seeing this near the end of the errors file: Traceback (most recent call last): File "/var/lib/condor/execute/dir_4162136/nwtenv/bin/hyperopt-mongo-worker", line 10, in sys.exit(main()) File "/var/lib/condor/execute/dir_4162136/nwtenv/lib/python3.8/site-packages/hyperopt/mongoexp.py", line 1328, in main sys.exit(main_worker()) File "/var/lib/condor/execute/dir_4162136/nwtenv/lib/python3.8/site-packages/hyperopt/mongoexp.py", line 1410, in main_worker return main_worker_helper(options, args) File "/var/lib/condor/execute/dir_4162136/nwtenv/lib/python3.8/site-packages/hyperopt/mongoexp.py", line 1321, in main_worker_helper mworker.run_one(reserve_timeout=float(options.reserve_timeout)) File "/var/lib/condor/execute/dir_4162136/nwtenv/lib/python3.8/site-packages/hyperopt/mongoexp.py", line 1041, in run_one raise ReserveTimeout() hyperopt.mongoexp.ReserveTimeout

maxwnewcomer commented 3 years ago

Ahhh, so the reserveTimeout()'s are the condor machine trying to reach the database and ask for a job, but they don't get any response. There a two likely scenarios: We had two jobs running at the same time, and combined they were using more than 375 workers, or the master script wasn't ran with "screen" and ended up closing after some time. The early error runs shouldn't be an issue as they are NWTOPT trying to grab some information from the search space

maxwnewcomer commented 3 years ago

Were both of you using NWTOPT at the same time? What ip, port combo did you use for the database (ik you used 130.11.177.99:27020, but what about mike)? How many machines did both of you start up? And did you both use screen to run all of the main processes?

maxwnewcomer commented 3 years ago

In my experimentation each database can have at most ~750 connections. Each worker takes up two connections, the nwtopt master script takes two, and pull nwts takes two, but disconnects and reconnects frequently

hwreeves-USGS commented 3 years ago

I used screen for both the mongodb and optimize_NWT commands (2 shells, each with screens).

I ran on port 27020 and asked for 300 nodes - which seemed to work OK looking at condor_q

HWR

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 8:58 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Were both of you using NWTOPT at the same time? What ip, port combo did you use for the database? How many machines did both of you start up? And did you both use screen to run all of the main processes?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924904707&data=04%7C01%7Chwreeves%40usgs.gov%7Cc933a51719ec4c4ad96708d97dc8a686%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679123031073545%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=9UKe9LVg%2BYeq3wTe30bpwIB2yShHjdQ3ZQmnzVk1gJE%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHEL5AM7S42KQYZHMYNPDUDHHFTANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7Cc933a51719ec4c4ad96708d97dc8a686%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679123031083503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8or3kHR3tDAKvH7THdhCJfwooCEhP%2FpybXNOEV15qjM%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7Cc933a51719ec4c4ad96708d97dc8a686%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679123031083503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=KtisIpC7eJVpu1NwTkOH%2Be%2Fyu9%2FB09beMwKNsk6ApNw%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7Cc933a51719ec4c4ad96708d97dc8a686%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679123031083503%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bZIpphzI3Tvget0VUWm0n0bw1kqrFLy%2B9fpVFoecOjs%3D&reserved=0.

maxwnewcomer commented 3 years ago

Also could've been that you set your trials too low and nwtopt actually finished running and failed to populate the database with more trials. Could you send your nwt_performance.csv?

maxwnewcomer commented 3 years ago

Was mike running at the same time?

maxwnewcomer commented 3 years ago

What happens if you try to start 50 workers with the same key and an extra 100 trials?

hwreeves-USGS commented 3 years ago

yes- HWR

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 9:02 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Was mike running at the same time?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924908403&data=04%7C01%7Chwreeves%40usgs.gov%7Ccd8b0daafb19411ca46308d97dc95a73%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679126025632789%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=3vkc03wyhjqFn2ugQi0yfOd37RVLUfQoF%2Fc5JPAnHqY%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHEL3I3ZWSKG3SRRFBZPTUDHHXXANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7Ccd8b0daafb19411ca46308d97dc95a73%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679126025642745%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=JFtQENV7lIyIzxw%2BTfuSptf8Xci2jswxeCuhDX8uiGA%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7Ccd8b0daafb19411ca46308d97dc95a73%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679126025642745%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8ZoQID4vXeVeXL0Ri4dbthBL2S4h2h67j%2Bho3sQIh7A%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7Ccd8b0daafb19411ca46308d97dc95a73%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679126025642745%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Bc7vSUtCpmTwJ6468GXU22Squ6gLKgJdIuxUObHKGVs%3D&reserved=0.

maxwnewcomer commented 3 years ago

I haven't really tested concurrent users running multiple runs on the same machine, but likely there were too many overall connections between the both of you that clogged down the database. Unfortunately, when this happens, I have found that mongodb acts like Pythagoras's cup and purges all runs. Just to reiterate, reserveTimeouts() are all mongodb and nwtwork related issues.

hwreeves-USGS commented 3 years ago

I can find the nwt_performance.csv -- where would that be?

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 9:02 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Also could've been that you set your trials too low and nwtopt actually finished running and failed to populate the database with more trials. Could you send your nwt_performance.csv?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924908128&data=04%7C01%7Chwreeves%40usgs.gov%7C0909dfae1f47438d628508d97dc94b16%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679125781047266%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MYOQbFFSzILeCxEBE%2FhhzoFMHnplbuqIt3egfL5iJyI%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHEL7DRPSFO5ZIQR4ARTLUDHHWRANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7C0909dfae1f47438d628508d97dc94b16%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679125781057221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1edwF04jm8lXWQ8X4GcqDTEVCR3C3rCGnS0xgNPEPOU%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7C0909dfae1f47438d628508d97dc94b16%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679125781057221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1SagqvtcbXW9LmFfL1LHVKejivQgnBTw11mI%2BMxVJ18%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7C0909dfae1f47438d628508d97dc94b16%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679125781067179%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=T4z7Xty9eHACSc69WwLCgZzZN9k462E35Olip1lfoao%3D&reserved=0.

maxwnewcomer commented 3 years ago

You should just run python pull_nwts.py --ip 130.11.177.99 --port 27020 --key your_key (and if you were still running i would recommend --loop True)

Also, if @mnfienen could provide an update as to what his run ended up looking like, that would be great.

maxwnewcomer commented 3 years ago

I personally really like running pull_nwts during runs because it really helps you see deep down what each machine is returning

hwreeves-USGS commented 3 years ago

is there a way to list valid keys?

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 9:12 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

You should just run python pull_nwts.py --ip 130.11.177.99 --port 27020 --key your_key (and if you were still running i would recommend --loop True)

Also, if @mnfienenhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmnfienen&data=04%7C01%7Chwreeves%40usgs.gov%7C6740781c6ced4b8c1ee408d97dcaabf9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679131711552601%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XBBJGQ7qB63JYXygZaTIbQF0z9Ic5bp4pX1TN5R6wUQ%3D&reserved=0 could provide an update as to what his run ended up looking like, that would be great.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924917158&data=04%7C01%7Chwreeves%40usgs.gov%7C6740781c6ced4b8c1ee408d97dcaabf9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679131711552601%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FaaGVaFHwVVaubvmZnwcHRRqzcWkS1s%2FzFm4bwnVHQ0%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHELZNE4SMKGADAWEP3VLUDHI35ANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7C6740781c6ced4b8c1ee408d97dcaabf9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679131711562557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=zUwQ5zd7lPh0P0LtLpYQDbsuC2FoOdcQMAp%2BtRvUdvg%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7C6740781c6ced4b8c1ee408d97dcaabf9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679131711562557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NqEn955i0lpnVoCqYcko3g4yvjPuaZEfV3gCgEv45Fo%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7C6740781c6ced4b8c1ee408d97dcaabf9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679131711562557%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2mw%2F3VcdDBP3oBKKAAEZjJ7sTHxOtbHM9Ys6iK376LE%3D&reserved=0.

maxwnewcomer commented 3 years ago

There currently isn't, but sometimes using the up arrow in the terminal will get you to the command you used to start up NWTOPT which should have the key. That is a good idea for a feature though, I'll add an issue to have that setup

hwreeves-USGS commented 3 years ago

in order to run pull_nwts - mongodb needs to be running?

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 9:20 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

I personally really like running pull_nwts during runs because it really helps you see deep down what each machine is returning

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924924655&data=04%7C01%7Chwreeves%40usgs.gov%7Ce1335b2a6eaf4a098dec08d97dcbc382%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679136392508155%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rlGgcKUnmNDDcUadvKyU4Q5%2BdBd5XWDWruOa7Sn4V1s%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHEL6L7BXUVX3PMQQVYNDUDHJZLANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7Ce1335b2a6eaf4a098dec08d97dcbc382%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679136392518110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Zw%2F%2Bwr9x568OaMUcQ50mTLyA4djg9JpYPhVv8LPOFjA%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7Ce1335b2a6eaf4a098dec08d97dcbc382%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679136392518110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Zv0Z%2BvEL5W26gsBoYkKau8DEQ2evFgCRzznROAxVFHM%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7Ce1335b2a6eaf4a098dec08d97dcbc382%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679136392518110%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2vpVhmbaL1BWkntqY%2BptHWcRh9ZDF0bLqQj%2BZ3gvrtA%3D&reserved=0.

maxwnewcomer commented 3 years ago

Yep, because it's pulling directly from the database and needs to make a connection to request the information

hwreeves-USGS commented 3 years ago

here is the performance.csv file --- all 9999s

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 10:13 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Yep, because it's pulling directly from the database and needs to make a connection to request the information

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924972094&data=04%7C01%7Chwreeves%40usgs.gov%7C43b19b85eea649ac45ef08d97dd33623%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679168371783430%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=V4uDm%2FLrQLTmwYmqJMDHwe9CRNGXEjQU1fZ%2FhpoNz5k%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHEL7JWZG45BMSR2ABTKTUDHQA7ANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7C43b19b85eea649ac45ef08d97dd33623%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679168371793390%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=RsnVblxZwOFYennrijlrM7Cz0vLJF5GoVy7r1xRjIas%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7C43b19b85eea649ac45ef08d97dd33623%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679168371793390%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qtLQlpAt%2Fng0l4WhnYlq4eUgmhC80HQamzd8%2Bo%2BGbcw%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7C43b19b85eea649ac45ef08d97dd33623%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679168371793390%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oQh67YszxFfrobN9YREsPnfkFbd6j%2FybLusTaByO5T0%3D&reserved=0.

maxwnewcomer commented 3 years ago

How many trials?

maxwnewcomer commented 3 years ago

Are there gaps in the NWTNUMs? like 2,3,5,8,12?

hwreeves-USGS commented 3 years ago

I submitted 300; but stopped the run after a little while when I saw that the list was all 999s

I could try something like 20 trails with 10 workers if that will help.

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 10:28 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

How many trials?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924985735&data=04%7C01%7Chwreeves%40usgs.gov%7C1925e9021df04849962f08d97dd550d7%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177412562748%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=IrkzVdCqJolhbZiBOHcN%2FDUCj%2FYOUvrAUd7l%2FuUv3eU%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHELYSW2ADNSMHDIHOQYLUDHRZXANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7C1925e9021df04849962f08d97dd550d7%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177412562748%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=PJaoY%2BavInwk7jkTA3R3hRYcQZB8Y0Pr7HP1Pd9JUaw%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7C1925e9021df04849962f08d97dd550d7%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177412562748%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=H2B6uNdGcJ9GUOkEouoS4bjN9S5Sp%2Flx%2BXfYIgUVXaM%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7C1925e9021df04849962f08d97dd550d7%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177412572703%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Nq2eyf0IzELfOyWXY%2B7y0PGPRLWIl%2FLl4wWVZPB%2BcEY%3D&reserved=0.

hwreeves-USGS commented 3 years ago

I don't see any gaps in the numbering of the nwt files

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 10:29 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Are there gaps in the NWTNUMs? like 2,3,5,8,12?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924986094&data=04%7C01%7Chwreeves%40usgs.gov%7C6104d1c8c13447e49ba008d97dd562e2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177715082721%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BoaFkgGSp%2F1SOSTjW2N%2BSoIY9Eb4INltuCL5Qrd1KjM%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHELZAYCTB3QLHK2Z4SLLUDHR3HANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7C6104d1c8c13447e49ba008d97dd562e2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177715092672%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=D9dX49gnm2eFXiloI9ewwnus0XevUvRRxlK11rKbg%2BM%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7C6104d1c8c13447e49ba008d97dd562e2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177715092672%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UXzOtriMvnIKOXl95ioXrl5qQP3py%2Fv8x02%2FZzQHMX0%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7C6104d1c8c13447e49ba008d97dd562e2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177715102627%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QrhWbnmRnriew0UaPoX%2FwRmoydySBn34HI%2BYjtCGS2o%3D&reserved=0.

hwreeves-USGS commented 3 years ago

shoot, sorry for so many short emails.

One thing I forgot to mention is that Mike and I are running the linux compiled version of mfnwt --> not using wine and the windows version.

I wonder if that is why mfnwt is starting OK but then nwtopt says it is finished so quickly??

HWR

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Reeves, Howard W @.> Sent: Wednesday, September 22, 2021 10:40 AM To: maxwnewcomer/NWTOPT @.>; maxwnewcomer/NWTOPT @.> Cc: Mention @.> Subject: Re: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

I don't see any gaps in the numbering of the nwt files

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 10:29 AM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Are there gaps in the NWTNUMs? like 2,3,5,8,12?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-924986094&data=04%7C01%7Chwreeves%40usgs.gov%7C6104d1c8c13447e49ba008d97dd562e2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177715082721%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=BoaFkgGSp%2F1SOSTjW2N%2BSoIY9Eb4INltuCL5Qrd1KjM%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHELZAYCTB3QLHK2Z4SLLUDHR3HANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7C6104d1c8c13447e49ba008d97dd562e2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177715092672%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=D9dX49gnm2eFXiloI9ewwnus0XevUvRRxlK11rKbg%2BM%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7C6104d1c8c13447e49ba008d97dd562e2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177715092672%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UXzOtriMvnIKOXl95ioXrl5qQP3py%2Fv8x02%2FZzQHMX0%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7C6104d1c8c13447e49ba008d97dd562e2%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679177715102627%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QrhWbnmRnriew0UaPoX%2FwRmoydySBn34HI%2BYjtCGS2o%3D&reserved=0.

maxwnewcomer commented 3 years ago

Would you mind pointing me to where your model is and I'll try running it on my own?

hwreeves-USGS commented 3 years ago

it's on M000

hwreeves/NWTOPT has the run.sh and nwtopt.sub; I put port = 27020 in there and used it in my run lines but I guess you can use your sub file and make the port match when you call mongodb and optimize_NWT

Project files in

hwreeves/NWTOPT/NWT_SUBMIT/PROJECT_FILES

The executable is 'mfnwt' and it is in the PROJECT_FILES directory.

Howard W. Reeves Acting Chief Science Officer, Groundwater and Watershed Modeling Team Lead USGS Upper Midwest Water Science Center 5840 Enterprise Drive Lansing, Michigan 48911 517-887-8914


From: Max Newcomer @.> Sent: Wednesday, September 22, 2021 2:16 PM To: maxwnewcomer/NWTOPT @.> Cc: Reeves, Howard W @.>; Mention @.> Subject: [EXTERNAL] Re: [maxwnewcomer/NWTOPT] timeout units (#15)

This email has been received from outside of DOI - Use caution before clicking on links, opening attachments, or responding.

Would you mind pointing me to where your model is and I'll try running it on my own?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmaxwnewcomer%2FNWTOPT%2Fissues%2F15%23issuecomment-925169338&data=04%7C01%7Chwreeves%40usgs.gov%7C7452c06dcd0e43c6e60008d97df525f9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679314133021717%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ivGNDkr7fzcegdi9Fu4Xdq1YV5%2F170meUgnvQc2UAK0%3D&reserved=0, or unsubscribehttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABMHEL6LYTCQSNDXBYPOA3TUDIMPNANCNFSM5EQBBRDQ&data=04%7C01%7Chwreeves%40usgs.gov%7C7452c06dcd0e43c6e60008d97df525f9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679314133021717%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Nrr1xTfOBg7hKbiijmlfgHXaeSEaXxBPEy%2BZr2dGnkU%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Chwreeves%40usgs.gov%7C7452c06dcd0e43c6e60008d97df525f9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679314133031672%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=IIY3opfHqe3FL2kklKailCxFNSH%2BPHa9jObZSsGEh7k%3D&reserved=0 or Androidhttps://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Chwreeves%40usgs.gov%7C7452c06dcd0e43c6e60008d97df525f9%7C0693b5ba4b184d7b9341f32f400a5494%7C0%7C0%7C637679314133031672%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=75K3KzPV%2FFsEjuvkP3ssjDkMKgF7RCxf1zaBZNx3C28%3D&reserved=0.

maxwnewcomer commented 3 years ago

Great thanks, I'll give it a try tonight. If you and @mnfienen could just make sure that you don't have your own mongodb process running by tonight that would be great

maxwnewcomer commented 3 years ago

I'm going to close this issue since it seems that everything including the timeout is working correctly and that Howard's run seemed to be using a model that has some issues. Constant ReserveTimeout and SeverSelectionErrors are able to be fixed by changing database port. If anyone disagrees, I'll be happen to reopen it.