benchmark code of four navigation algorithms

[nina124]

Hi, Will the benchmark code of four navigation algorithms in the paper be released?

Also, how long does it take to train the agents? My english is poor, so I have some confusion about the following sentences in paper

The agents are trained and tested over episodes lasting up to 500 time steps, with 10 steps per second of simulated time. Each agent is trained for a total of 13.2Mtime steps, corresponding to roughly 15 days of experience. Average training speed with four simulation threads is about 167 steps per second, amounting to approximately 14.4M steps per day. We run four such training processes (four simulation threads each) on a single Nvidia Titan X Pascal GPU, yielding a total of 57.6M steps per day (668 steps per second).

My confusion is What the training speed is? The above words mentioned two kinds of speed: "10 steps per second" or "668 steps per second with four simulation threads"? What are the differences between them?

Thanks.

[msavva]

Hi @nina124 ,

The benchmark code for three out of four of the navigation algorithms is released at https://github.com/minosworld/unreal (A3C, A3C-LSTM and UNREAL), while the DFP baseline is at https://github.com/minosworld/dfp . Please let us know if you have any issues running the baselines (the latter one in particular is a somewhat preliminary release).

With regards to training speed: the 668 steps per second is the effective speed when running four experiments with four simulation threads each on a single Titan X Pascal GPU. The total training time for a given experiment varies somewhat depending on the specifics of the environment and the modalities used. Note that the "10 steps per second" refers to the time within the simulation itself (i.e. how much time elapses inside the simulation per agent step, or equivalently how many agent steps there are in a simulation second).

[nina124]

Thanks @msavva I will try these baselines. Another question. Now I successfully run minos on headless server, but without gpu rendering. For faster rendering speed, Is it possible to render with GPU on headless server, like using virgualgl or turbovnc? I just found the instructions on "Run CARLA without display and selecting GPUs". I guess it might offer some hint on this. But I have no idea about these area.

[msavva]

Hi @nina124 ,

We have some instructions for running headless (see https://github.com/minosworld/minos/blob/master/FAQ.md#how-can-i-run-headless-on-a-server)

Using VirtualGL + TurboVNC is also supported but we don't have instructions here. Actually, can you summarize the steps you took to run headless without gpu rendering? They would be helpful for others who want to run on machines with no gpu resources.

[nina124]

Hi @msavva I tried the unreal baseline with python3 main.py --env_type indoor --env_name pointgoal_suncg_se --parallel_size 10. But the agent failed to learn something meaningful. The tensorboard score is shown below.

Did your team use the default hyperparameter settings(flags in options.py) when training all the three environments in the following? python3 main.py --env_type indoor --env_name pointgoal_suncg_se python3 main.py --env_type indoor --env_name objectgoal_suncg_mf python3 main.py --env_type indoor --env_name roomgoal_mp3d_s

And, could you share the learning curve of score(the tensorboard result)?

[kvas7andy]

As I am not sure about pointgoal task, I can't state anything here, except that there is another issue #38, which can shed light on this one.

[nina124]

@kvas7andy Thanks, I have commented on that issue and hope I can get some useful training experience.