BUG: fail in call to train_model from train_network.py when a path_model_init is provided

[oumayb]

I'm trying to use train_model with providing initial weights, but I'm having several issues.

• The first one: IOError: [Errno 2] No such file or directory: '/Users/oubou_admin/axondeepseg/AxonDeepSeg/models/default_TEM_model_v1/evolution.pkl' That's because I don't have an evolution file for the model I'm loading the weights from, but I think the function should run even if there is no such file provided. What are your thoughts on that? I temporarily removed it locally to be able to run the rest of the code, but then I had:

• The seconde one: KeyError: 'boundaries_activate', caused by this line: https://github.com/neuropoly/axondeepseg/blob/2def1c0a022b009ba9c81ae96e9b3f36ab66f150/AxonDeepSeg/data_management/input_data.py#L406 I can't think of any temporary fix for this one

[mathieuboudreau]

@oumayb Regarding the missing pkl file, it was never commited because there used to be a gitignore rule here that I removed recently. If @alzaia kept the saved model on a computer, that file may still be there locally for the model you mentioned above, I just don't know which one. Maybe @alzaia would know? I don't know how necessary this file is for what you want to do, I'm not experienced/familiar with running training models with initial weights to know.

For your second issue, it seems that weifhts_modifier is created by the function argument here:

https://github.com/neuropoly/axondeepseg/blob/2def1c0a022b009ba9c81ae96e9b3f36ab66f150/AxonDeepSeg/data_management/input_data.py#L244-L247

which sets it to false by default, but can be set in the function call here:

https://github.com/neuropoly/axondeepseg/blob/1edb7dd5bb3ccc2e1488015a3401b17b9b480cfd/AxonDeepSeg/train_network.py#L365-L370

and/or here:

https://github.com/neuropoly/axondeepseg/blob/1edb7dd5bb3ccc2e1488015a3401b17b9b480cfd/AxonDeepSeg/train_network.py#L449-L455

where the parameter weights_modifier is defined by using the config file here:

https://github.com/neuropoly/axondeepseg/blob/1edb7dd5bb3ccc2e1488015a3401b17b9b480cfd/AxonDeepSeg/train_network.py#L53

which is defined here:

https://github.com/neuropoly/axondeepseg/blob/b3d9574d06739c7e52d2ce832b5efa90031a5f69/AxonDeepSeg/train_network_tools.py#L50-L65

So maybe try defining a key called weighted_cost-boundaries_activate with a value of True in the config network file?

There's also a json file in this repo that has a boundaries_activate key, but it doesn't seem like it would work like above:

https://github.com/neuropoly/axondeepseg/blob/5078fc9019520f619fb75f62c863faab19239af5/AxonDeepSeg/models/default_TEM_model_v1/config_network_original.json#L12-L21

[mathieuboudreau]

On a side note, this line here:

https://github.com/neuropoly/axondeepseg/blob/b3d9574d06739c7e52d2ce832b5efa90031a5f69/AxonDeepSeg/train_network_tools.py#L59

might be creating a bug, since @alzaia changed that key to weighted_cost-activate, reinforcing my worries about ensuring retrocompatibility of config files, re: #114

[mathieuboudreau]

Oh and I just fully understood your first question. I don't know why the pkl file should be needed by default for that case, I'd ask @alzaia if this is something that could be omitted permanently or if there's an advantage to it.

[oumayb]

Thanks @mathieuboudreau, I think in fact the bug is caused by the way the variables are written in the config file I'm using. I'm making the following changes:

• weighted_cost_boundaries_sigma==> weighted_cost-boundaries_sigma
weighted_cost_boundaries_activate ==> weighted_cost-boundaries_activate which is different than what I told you yesterday, I'm sorry about that! https://github.com/neuropoly/axondeepseg/issues/114

Also, should we fix this line? https://github.com/neuropoly/axondeepseg/blob/b3d9574d06739c7e52d2ce832b5efa90031a5f69/AxonDeepSeg/train_network_tools.py#L59 and replace weighted_cost_activate with weighted_cost-activate?

As for the pkl file, I think it can be omitted permanently but let's wait for @alzaia 's confirmation.

[mathieuboudreau]

@oumayb Ah ha, that change makes sense now, and so does the bug!

I'm fairly sure that bug should be fixed, but would need to look at the code a bit more just to be 100% sure.

[alzaia]

Hey guys, I'm not sure I remember what the pkl file was for. But if you run the training using the notebook, it will also create a config file (that should be valid) and the files TF uses for training/save/restore, so you can check which files are expected to be there. I can look more into it this weekend.

[oumayb]

@alzaia thanks! For the pkl, I'm currently training a model initialized with pre-trained weights, without providing a pkl evolution file, I'll see if it causes any problem.

[mathieuboudreau]

Closing, files don't exist anymore since our move to IVADOMED.