Running Issues

[s3afroze]

Hi, I recently started to get this new user warning in Heuristics Developer:

/home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:1454: UserWarning: Cannot decide type for the following arguments. Consider providing them as input:
    data0: None
  input_sym_arg_type = in_param.infer_type()[0]
/home/ec2-user/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet/gluon/block.py:1454: UserWarning: Cannot decide type for the following arguments. Consider providing them as input:
    data0: None
  input_sym_arg_type = in_param.infer_type()[0]

Everything else was running as expected by then suddenly when I run the next cell recently i.e. "Simulation loop", it prints "completed" twice while it used to print only once before.

When I tried running the SageModelTraining Notebook, I get the following error:

RuntimeError: Failed to run: ['docker-compose', '-f', '/tmp/tmpic2b7kee/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1

Sorry for the long message and I would really really appreciate your guidance!

[s3afroze]

When I restarted the notebooks, the Heuristic Developer notebook started to work properly without any issues but I have the same issue with the Traning Notebook

I upgraded pip and installed Twilio package before I trained the models - not sure if that's the reason or not.

[jonomon]

Did you get this error after running the endpoint creation?

RuntimeError: Failed to run: ['docker-compose', '-f', '/tmp/tmpic2b7kee/docker-compose.yaml', 'up', '--build', '--abort-on-container-exit'], Process exited with code: 1

This error occurs if you try to build an endpoint locally when there is already an endpoint. To delete the previous local endpoint, you'll need to stop the docker container.

Firstly run and this will print all the containers: !docker ps -a

Get the container ID and run:

!docker stop {containerid}

[s3afroze]

Thank you for your quick response, Jonathan! I ran that cell when I deployed the previous model and was not really using the Training notebook until now to train new changes so it could be but I am not sure.

I followed the instructions but I am still getting the same error - I will restart the notebook and then try again as well.

[s3afroze]

Hi Jonathan, I realized that it is because there was an unbound local error - it's interesting that I did not get the error when I was testing it in the Heuristics Development model notebook. The docker error showed up after the local error so I was not reading the problem correctly.

[s3afroze]

I am opening temporarily because I am interested in knowing if you have any thoughts on why the problem did not occur while running the Heuristics Development model notebook.

Probably a silly question but I wanted to confirm if we are supposed to run Hyperparameter optimization to tune after training or is it training by itself so no need to run the training cell?

[jonomon]

You run hyperparamter optimization (HPO) when you are trying to find a set of parameters for your problem. So it serves a similar but different purposes compared to training.

I'm not too sure I follow your problems with the heuristics development. Could you send a message on the "snake-dev-machine-learning" in the official battlesnake channel and we can discuss there?