Closed vendetta1987 closed 5 years ago
mikkokotila
commented
5 years ago I checked the data and the error message, but it appears that there are 18 columns (and not 20 as the error message says in the provided trace). The fact that the column has NaN does not cause the error you have in the trace. Or did I miss something?
vendetta1987
commented
5 years ago Thank you for picking this up.The problem seems to stem from the fact that Talos is expecting the amount of columns the first run returned. If it was successful the table will contain validation data columns. Of it wasn't those will be missing from the header. Upon the next run the CSV will be read and if that run had a different outcome its amount of result columns will be different. Internally Pandas will then throw the given error.This happens with interleaved NaN/Inf runs especially.
mikkokotila
commented
5 years ago Thanks for clarification. Can you share with me the output csv you get where this error is reported.
vendetta1987
commented
5 years ago It's the same I used when first posting. Looking at the last two lines you'll notice those actually contain two additional column entries, both validation results.
mikkokotila
commented
5 years ago Can you try:
pip install -U git+https://github.com/autonomio/talos@params-api-test
This installs the next major release (v.0.6) where logging has been to an important part rebuilt.
vendetta1987
commented
5 years ago I now updated to the branch and fixed the errors that popped up. Seems some API changes will need to be updated in the documentation. I let it run for a weekend and it was still running upon my return. Although there were no NaN results, those were fixed elsewhere, the new Scan() method doesn't seem to do random testing anymore. The following code will happily check all possible parameter combinations in order. There is no random choise of combinations to cover more ground more quickly.
ta.Scan(dummy[0], dummy[1], talosParams, modelFncTalos, random_method="quantum", clear_session=True, reduction_method="correlation", reduction_metric="val_loss", minimize_loss=True, reduction_interval=5, reduction_window=5, print_params=True, experiment_name=TALOS_LOG_NAME+"_"+str(expNr))
What needs to be changed in the call to get the random sampling back? I do like the fact it doesn't create all combinations at the beginning while hogging RAM though.
mikkokotila
commented
5 years ago What do you mean by not doing random testing anymore? In your case, you have chosen to use quantum random method, so that will be used for pick n permutations. But I can see that you did not invoke fraction_limit so you will get all the permutations in your parameter space.
mikkokotila
commented
5 years ago Closing here. Feel free to open new issue if anything. Thanks and have a great day too! :)
I noticed my talos scans crashing recently and found the following stack trace:
Indeed the CSV report doesn't contain validation values for some runs that ended in NaN or Inf loss values. This seems to be related but not limited to the usage of the
TerminateOnNaN()callback provided with keras. I thought the callback was quite helpful to shorten scanning times but if the first run gets terminated automatically, the resulting CSV report lacks the necessary columns to continue:Maybe talos could check whether all data is available and fill gaps with something meaningful. It shouldn't crash at least.