Open wkitlasten opened 1 month ago
Update, I'm even more confused, shocking I know. Suspecting the _
was somehow being replace by a decimal in the mlts.index I changed the name, but I get the same errror. But in this case they seem to be aligned?
AssertionError: Probable miss-alignment in tpl indices and original file:
mult idx[:10] : ['123n0', '123n1', '180n0', '180n1', '180n10', '180n11', '180n12', '180n13', '180n14', '180n15']
org file idx[:10]: ['123n0', '123n1', '180n0', '180n1', '180n10', '180n11', '180n12', '180n13', '180n14', '180n15']
n common: 576, n cols: 1, expected: 0.0.
Next step, made the index an int, same issue. I doubt I could misunderstand what is going on more! They look aligned to me. I think the only thing left is the sep?
AssertionError: Probable miss-alignment in tpl indices and original file:
mult idx[:10] : [1230000, 1230001, 1800000, 1800001, 1800002, 1800003, 1800004, 1800004, 1800006, 1800007]
org file idx[:10]: [1230000, 1230001, 1800000, 1800001, 1800002, 1800003, 1800004, 1800004, 1800006, 1800007]
n common: 518, n cols: 1, expected: 0.0.
Got it to work without any use_rows
arg and grid scale, not constant as intended.
Seems there is a few things here @wkitlasten.
-1 shifting
This is a consequence of the zero_based
arg passed to the parent pstfrom object, it is the same as when you index by k,i,j. It should be safe for pyemu alignment at runtime -- you just need to remember to add the 1
back in any post processing. There could be an argument for supporting a modification of this arg per add_parameters()
and add_observations()
calls -- but this is also not without it's drawbacks, not least keeping track of which pars and obs are zero-based.
mfile_skip
This just tells pyemu how many initial model input file line to skip when reading. If your headers are sane (same delimiter and same number of cols as the rest of the file) you shouldn't need it and you can pass index_cols='ifno'
directly (you will still get the -1 adjustment to the integer index values, as above).
use_rows
There are some hints in the docs(!) (unexpected, I know!). So if Theo object passed to use_rows
has a single dimension (e.g. [0, 4, 7]
) it is interpreted as a locational (kinda like pandas iloc
). If you want to match on index col values (as they are in the model input file you can pass an object with 2 dimensions. This feels a bit weird of cases like yours where you only have one index_col, but for you it might be something like [(1,), (1109,), (1110,), ...]
-- this should behave a little more like pandas .loc
. Note: when passing this way the values are expected to be in the model base (so actual index col values in the model input file, prior to any -1 adjustment)
misalignment This one looks like it might need a bit more digging -- if it is still not working after checking the other points above, feel free send thought that file and I can take a look.
Thanks for that. The main issue was the use_rows = [(1,)...] bit and inadvertantly mixing positional with label indexes (which confounded the first two).
I read the hints in the docs but it just wouldn't penetrate my thick skull. Not that it makes any more sense that what is written, but perhaps a more explicit explanation could be added to the docstring? Something like:
"For use_rows with a single 'index_cols' use [('a',),('b',),('c',)] to set parameters for rows with model file index entries of a,b,c."
This issue arises from line 2099 of utils.helpers() when index names have _
in them. Apparently 123_1 is a "convenient" way to write 1231.0 (e.g., https://stackoverflow.com/questions/54009778/what-do-underscores-in-a-number-mean).
Probably a more explicit check for non-numeric parts of the index (that are not .
) is in order?
Hey,
Can you point me to a simple example that uses
par_type="constant"
anduse_rows=[list-o-lines]
within pf.add_parameters() so I can break it?! A bit below, but I will spare anyone else the pain of looking at the rest of my mess!I am using this to parameterize all lines in some files and it seems to work mostly as expected:
The file formerly known as fname (.csv):
If I leave
mfile_skip=1
out it tries to parameterize the header (e.g.,parnme = pname:sfr.inflow_inst:0_ptype:gr_usecol:2_pstyle:m_idx0:ifno
which is obviously wrong!If I have
mfile_skip=1
in the callifno
is shifted by -1 from the original model input file (e.g.parnme = pname:sfr.inflow_inst:1_ptype:gr_usecol:2_pstyle:m_idx0:0
). If I leavemfile_skip
out of the call and use index labels rather than locations for use_cols and index_cols I get a similar result (ifno shifted -1). I don't understand this, but maybe that is okay, it ain't the first time. That produces the following .tpl file.I do something similar for other files, but with a subset of lines. It seems the use_rows arg requires positional indexing, hence I use positional for use_cols and index_cols too and require the mfile_skip=1 to avoid parameterizing the header:
That builds the following .tpl file, from the original file which is as expected.
Pest builds fine from pst_from with the above bits. But when I try to
pyemu.helpers.apply_list_and_array_pars(arr_par_file='mult2model_info.csv')
I get the following error related to the 2ndadd_parameters
call:From line 2213 in helpers.py
So clearly the issue is in
common_idx = (new_df.index.intersection(mlts.index).drop_duplicates())
but I can't track how mlts gets its index or how I can adjust my pf builds to ensure it lines up with "new_df".Any suggestions would be helpful.