No Visdom visualization is created, because the file that should contain the data that should be used is never created. You can specify the location where this file should be store, and this variable is passed to make_env function, but it's not used there.
It seems that something is missing.
I took the RL code from here and modified it to eliminate dependencies on OpenAI baselines. You could go back to this original code and apply the necessary modifications. Another option is to rewrite the Visdom code, which is not too difficult.
No Visdom visualization is created, because the file that should contain the data that should be used is never created. You can specify the location where this file should be store, and this variable is passed to make_env function, but it's not used there. It seems that something is missing.