search

USGS-R / river-dl

Deep learning model for predicting environmental variables on river systems
Creative Commons Zero v1.0 Universal
21 stars 14 forks source link

Try model without pretraining #38

Open jsadler2 opened 4 years ago

jsadler2 commented 4 years ago

I would like to try the model without pretraining on the PRMS/SNTemp. I'm curious to see just how bad it would do. I've also been thinking that we really gain a lot from having the process model and it'd be nice to put a number to that. I think having that number could be a good way to show to the process-modeling community that their work is critical to this type of modeling. We can't just rely on observations. We need good process models.

jdiaz4302 commented 3 years ago

You can assign this to me, I'm looking into it for a presentation for Alison. Current plan is to run an experiment of 5 replicates of (1) using process-based pretraining and (2) using process-based outputs as inputs to the RGCN without pretraining. Aiming for a test of extrapolation where we leave summer observations out of the training data, then evaluating with attention on performance by season (summer or not) - I believe you all have some precedent with this being a strongish test for extrapolation. Trying to get this done in the coming days to allow incorporation into her upcoming talk.

jsadler2 commented 3 years ago

sounds awesome! I'm really excited to see the results

jdiaz4302 commented 3 years ago

I have some results and was hoping to open a PR with the code that produced them for others to review, but I'm not allowed to push.

[jdiaz@tg-login1 river-dl] git push origin no-pretraining-exp
ERROR: Permission to USGS-R/river-dl.git denied to jdiaz4302.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The PR code is pretty ad-hoc, I don't intend or expect it to be merged into this clean, concise repo, but I want to document how we could do this experiment and the results that I got.

jdiaz4302 commented 3 years ago

Permissions issue is resolved now - PR shortly

aappling-usgs commented 3 years ago

@jdiaz4302 also consider pushing to a personal fork in the future, unless we've changed our general git flow and I missed it (looks like origin is set to USGS-R/river-dl above)

jdiaz4302 commented 3 years ago

Ah, okay. I'm used to pushing to an experimental branch within the origin repo - no problem adjusting though.

aappling-usgs commented 3 years ago

Sneaking a peek before you PR - wondering if the multiple training periods and/or the shift in start day of year from training to validation might create any issues (I hope the latter isn't the case, but I'm not sure).

As a diagnostic (or maybe a solution), could you try writing a different version of the obs file that has summer observations set to NA in the training period, then using that file to train? Then you could keep the same train, val, test period definitions as before. This might involve writing your own evaluation code, if the same obs file gets used for training and evaluation, but could get us around the potential issues with disjoint training periods.

SimonTopp commented 2 years ago

Just looking back through this issue. @jdiaz4302, did the results of your experiment ever get summarized anywhere other than the PR? I'd be curious to know where they live for reference.