search

flow-project / flow

Computational framework for reinforcement learning in traffic control
MIT License
1.06k stars 375 forks source link

Increasing the number of vehicles in Figure Eight scenario #522

Closed pnp91 closed 5 years ago

pnp91 commented 5 years ago

Choose an option This is a:

Describe the issue Figure eight scenario only accommodates a maximum of 14 vehicles in Flow. Even if I give more than this, it is not effective. But I am planning to extend this to 100 vehicles involving RL and human vehicles. I just want to know the reason behind this limitation to be able to apply in a real world scenario. Can I add another route with this and expand the capacity_ Like merging one figure eight and one ring wave scenario or merging two figure eight scenarios to increase its capacity.

Expected behavior Many vehicles in a single road track

Desktop (please complete the following information):

Additional context https://flow.readthedocs.io/en/latest/examples.html#figure-eight

eugenevinitsky commented 5 years ago

Yes, you can absolutely combine multiple scenarios or increase the length of the roadway to accommodate more vehicles. If you point me at the specific scenario we can try and figure out the exact change you would need to make.

pnp91 commented 5 years ago

@eugenevinitsky Thank you so much for your kind reply and I deeply regret and apologize for delay in my response. My main goal is to accommodate more autonomous vehicles in figure eight scenario. And not to restrict to 14 vehicles. A solution for this, I have identified to merge the scenarios inorder to increase the capacity of ring. I will definitely try increasing this and get back to you in-case I face further difficulties.

eugenevinitsky commented 5 years ago

Sounds good! You should also be able to increase the ring length parameter, which should give you more space to stick extra vehicles in.

pnp91 commented 5 years ago

Oh yes I am aware of this parameter but wasn't aware that it can hold extra vehicles in. Thank you so much for the solution. That sounds a good option. And especially thanks a lot for timely constant support and encouragement, explaining us with all patience without ignoring. I have noticed it many times and feel so happy for this pro-active support from Team Flow. Kudos!!

jingyanping commented 5 years ago

When I was running the flow/examples/rllib/stabilizing_the_ring.py. it generated a series of files, such as 'checkpoint_20', 'checkpoint_40'..., 'progress.csv', 'result.json'. When I finished training, I didn't know how to look at my training results. I read the documentation find that have two ways to resolve that,

First : input code python ./visualizer_rllib.py /ray_results/result_dir 1 at terminal, but this command is reported an error can't open file....

Second:tensorboard --logdir=~/ray_results, the terminal always displays "TensorBoard. 1.9.0 at http://ubuntu:6006 (Press CTRL+C to quit)" So I want to know how I should look at the results after my training.@pnp91@eugenevinitsky

pnp91 commented 5 years ago

@jingyanping The first command is used to visualize the files from any particular checkpoint number. But for this, you have to be in the path of Visualize file. You can make "cd flow/flow/visualize" and then view the output at one particular checkpoint. For example if you wish to see the output at checkpoint 40, then type the following after you are at visualize folder python ./visualizer_rllib.py /home//ray_results/<exp_tag(here for this example, it is "lord_of_numrings">/<number of checkpoint. Here it is"40"> In the above command, I have used < > for you to be editable and do not use with these brackets.

Second : It is for visualizing a graph for training steps, rewards, entropy and so on.. You can press on the link to open in a new tab and check the graph of results.

Hope it helps you. @eugenevinitsky : As I was also mentioned in this question and I know the answer, I have replied. I hope it should not be a problem.

jingyanping commented 5 years ago

Thank you very much for your kind reply, i followed your suggestion, ''' (flow) sean@ubuntu:~/flow/flow/visualize$ python ./visualizer_rllib.py /ray_results/stabilizing_the_ring/checkpoint 40 ''' But it has reported an error: ''' Traceback (most recent call last): File "./visualizer_rllib.py", line 384, in visualizer_rllib(args) File "./visualizer_rllib.py", line 57, in visualizer_rllib config = get_rllib_config(result_dir) File "/home/sean/flow/flow/utils/rllib.py", line 169, in get_rllib_config "Could not find params.json in either the checkpoint dir or " ValueError: Could not find params.json in either the checkpoint dir or its parent directory. '''

But in my ray_results, i find the file params.json. Can you give me some advice?

At 2019-07-18 15:24:46, "niyathipriya" notifications@github.com wrote:

@jingyanping The first command is used to visualize the files from any particular checkpoint number. But for this, you have to be in the path of Visualize file. You can make "cd flow/flow/visualize" and then view the output at one particular checkpoint. For example if you wish to see the output at checkpoint 40, then type the following after you are at visualize folder python ./visualizer_rllib.py /home//ray_results/<exp_tag(here for this example, it is "lord_of_numrings">/<number of checkpoint. Here it is"40"> In the above command, I have used < > for you to be editable and do not use with these brackets.

Second : It is for visualizing a graph for training steps, rewards, entropy and so on.. You can press on the link to open in a new tab and check the graph of results.

Hope it helps you. @eugenevinitsky : As I was also mentioned in this question and I know the answer, I have replied. I hope it should not be a problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

pnp91 commented 5 years ago

I think the problem is you used wrong name for .. It is not the name of .py file. But there is a parameter inside .py file called exp_tag, you have to copy that name. It is also the same name which you have inside your "ray_results" folder. One more problem is checkpoint number. You just need to mention the number and not as "checkpoint_40". Hopefully my below example screenshot helps you. Screenshot from 2019-07-18 09-56-05

jingyanping commented 5 years ago

oh yes, i resolve that problem follow your advice,Thank you very much!!!

pnp91 commented 5 years ago

Cool! 😀

On Thu, 18 Jul 2019 at 10:32, jingyanping notifications@github.com wrote:

oh yes, i resolve that problem follow your advice,Thank you very much!!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flow-project/flow/issues/522?email_source=notifications&email_token=AH6J3WKC6FRFPFEEBR7I6S3QAATB5A5CNFSM4HMZBNL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2HXZNA#issuecomment-512720052, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6J3WL7DCXPY6XQXS6LHT3QAATB5ANCNFSM4HMZBNLQ .

jingyanping commented 5 years ago

I'm sorry to disturb you again. About the "tensorboard-logdir=~/ray_results" command, Is it meaning that when i open one terminal to train the vehicle, and input "tensorboard-logdir=~/ray_results" in another terminal at the same time? If i do that, the terminal show that" TensorBoard 1.9. 0 at http://ubuntu:6006 (Press CTRL+C to quit)", but i can't open the Link, i think that is not right.

At 2019-07-18 15:24:46, "niyathipriya" notifications@github.com wrote:

@jingyanping The first command is used to visualize the files from any particular checkpoint number. But for this, you have to be in the path of Visualize file. You can make "cd flow/flow/visualize" and then view the output at one particular checkpoint. For example if you wish to see the output at checkpoint 40, then type the following after you are at visualize folder python ./visualizer_rllib.py /home//ray_results/<exp_tag(here for this example, it is "lord_of_numrings">/<number of checkpoint. Here it is"40"> In the above command, I have used < > for you to be editable and do not use with these brackets.

Second : It is for visualizing a graph for training steps, rewards, entropy and so on.. You can press on the link to open in a new tab and check the graph of results.

Hope it helps you. @eugenevinitsky : As I was also mentioned in this question and I know the answer, I have replied. I hope it should not be a problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

pnp91 commented 5 years ago

no.. you can do in any terminal after the training is completed.. you can right click on the link and open it in new tab. [image: Screenshot from 2019-07-18 14-00-38.png]

On Thu, 18 Jul 2019 at 13:55, jingyanping notifications@github.com wrote:

I'm sorry to disturb you again. About the "tensorboard-logdir=~/ray_results" command, Is it meaning that when i open one terminal to train the vehicle, and input "tensorboard-logdir=~/ray_results" in another terminal at the same time? If i do that, the terminal show that" TensorBoard 1.9. 0 at http://ubuntu:6006 (Press CTRL+C to quit)", but i can't open the Link, i think that is not right.

At 2019-07-18 15:24:46, "niyathipriya" notifications@github.com wrote:

@jingyanping The first command is used to visualize the files from any particular checkpoint number. But for this, you have to be in the path of Visualize file. You can make "cd flow/flow/visualize" and then view the output at one particular checkpoint. For example if you wish to see the output at checkpoint 40, then type the following after you are at visualize folder python ./visualizer_rllib.py /home//ray_results/<exp_tag(here for this example, it is "lord_of_numrings">/<number of checkpoint. Here it is"40"> In the above command, I have used < > for you to be editable and do not use with these brackets.

Second : It is for visualizing a graph for training steps, rewards, entropy and so on.. You can press on the link to open in a new tab and check the graph of results.

Hope it helps you. @eugenevinitsky : As I was also mentioned in this question and I know the answer, I have replied. I hope it should not be a problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flow-project/flow/issues/522?email_source=notifications&email_token=AH6J3WJZ5HJZMA74XBOUCRTQABK2LA5CNFSM4HMZBNL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2IHSSI#issuecomment-512784713, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6J3WNBATETCWFZ4TPX5XLQABK2LANCNFSM4HMZBNLQ .

jingyanping commented 5 years ago

Thanks!!! i will try it!

At 2019-07-18 20:02:36, "niyathipriya" notifications@github.com wrote: no.. you can do in any terminal after the training is completed.. you can right click on the link and open it in new tab. [image: Screenshot from 2019-07-18 14-00-38.png]

On Thu, 18 Jul 2019 at 13:55, jingyanping notifications@github.com wrote:

I'm sorry to disturb you again. About the "tensorboard-logdir=~/ray_results" command, Is it meaning that when i open one terminal to train the vehicle, and input "tensorboard-logdir=~/ray_results" in another terminal at the same time? If i do that, the terminal show that" TensorBoard 1.9. 0 at http://ubuntu:6006 (Press CTRL+C to quit)", but i can't open the Link, i think that is not right.

At 2019-07-18 15:24:46, "niyathipriya" notifications@github.com wrote:

@jingyanping The first command is used to visualize the files from any particular checkpoint number. But for this, you have to be in the path of Visualize file. You can make "cd flow/flow/visualize" and then view the output at one particular checkpoint. For example if you wish to see the output at checkpoint 40, then type the following after you are at visualize folder python ./visualizer_rllib.py /home//ray_results/<exp_tag(here for this example, it is "lord_of_numrings">/<number of checkpoint. Here it is"40"> In the above command, I have used < > for you to be editable and do not use with these brackets.

Second : It is for visualizing a graph for training steps, rewards, entropy and so on.. You can press on the link to open in a new tab and check the graph of results.

Hope it helps you. @eugenevinitsky : As I was also mentioned in this question and I know the answer, I have replied. I hope it should not be a problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flow-project/flow/issues/522?email_source=notifications&email_token=AH6J3WJZ5HJZMA74XBOUCRTQABK2LA5CNFSM4HMZBNL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2IHSSI#issuecomment-512784713, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6J3WNBATETCWFZ4TPX5XLQABK2LANCNFSM4HMZBNLQ .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jingyanping commented 5 years ago

hello， I'm sorry trouble you again. i have trained the auto car in a closed network using PPO algorithm, it can dissipate stop-and-go waves in the closed loop.(The PPO algorithm can also be replaced with other algorithm,such as A3C,DQN,DDPG,right?). i trained the auto car in closed network, Can it be migrated to an open-loop network,for example, on a single-lane road with one or two ramps? If so,can you give me some advice about how to do that? sincerely jing

At 2019-07-18 16:00:40, "niyathipriya" notifications@github.com wrote:

I think the problem is you used wrong name for .. It is not the name of .py file. But there is a parameter inside .py file called exp_tag, you have to copy that name. It is also the same name which you have inside your "ray_results" folder. One more problem is checkpoint number. You just need to mention the number and not as "checkpoint_40". Hopefully my below example screenshot helps you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

pnp91 commented 5 years ago

Hello Ping,

yes you can do that. You need to change in the scenario which you define with the specify routes, specify connections and specify edges functions. This is well explained further in tutorial on "how do we create our own custom scenarios". Although building a scenario is not a very easy task, I would suggest you to follow the tutorial given in FLOW. They have appreciable description which helps us to build our own tracks. Hope this helps.

On Thu, 25 Jul 2019 at 04:52, jingyanping notifications@github.com wrote:

hello， I'm sorry trouble you again. i have trained the auto car in a closed network using PPO algorithm, it can dissipate stop-and-go waves in the closed loop.(The PPO algorithm can also be replaced with other algorithm,such as A3C,DQN,DDPG,right?). i trained the auto car in closed network, Can it be migrated to an open-loop network,for example, on a single-lane road with one or two ramps? If so,can you give me some advice about how to do that? sincerely jing

At 2019-07-18 16:00:40, "niyathipriya" notifications@github.com wrote:

I think the problem is you used wrong name for .. It is not the name of .py file. But there is a parameter inside .py file called exp_tag, you have to copy that name. It is also the same name which you have inside your "ray_results" folder. One more problem is checkpoint number. You just need to mention the number and not as "checkpoint_40". Hopefully my below example screenshot helps you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flow-project/flow/issues/522?email_source=notifications&email_token=AH6J3WOENNUEJKMHJBV54VTQBEIORA5CNFSM4HMZBNL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2YFYLA#issuecomment-514874412, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6J3WKDRZUKBOQNZA3I4EDQBEIORANCNFSM4HMZBNLQ .

jingyanping commented 5 years ago

OK,thank you very much for your reply!

At 2019-07-25 16:05:24, "niyathipriya" notifications@github.com wrote: Hello Ping,

yes you can do that. You need to change in the scenario which you define with the specify routes, specify connections and specify edges functions. This is well explained further in tutorial on "how do we create our own custom scenarios". Although building a scenario is not a very easy task, I would suggest you to follow the tutorial given in FLOW. They have appreciable description which helps us to build our own tracks. Hope this helps.

On Thu, 25 Jul 2019 at 04:52, jingyanping notifications@github.com wrote:

hello， I'm sorry trouble you again. i have trained the auto car in a closed network using PPO algorithm, it can dissipate stop-and-go waves in the closed loop.(The PPO algorithm can also be replaced with other algorithm,such as A3C,DQN,DDPG,right?). i trained the auto car in closed network, Can it be migrated to an open-loop network,for example, on a single-lane road with one or two ramps? If so,can you give me some advice about how to do that? sincerely jing

At 2019-07-18 16:00:40, "niyathipriya" notifications@github.com wrote:

I think the problem is you used wrong name for .. It is not the name of .py file. But there is a parameter inside .py file called exp_tag, you have to copy that name. It is also the same name which you have inside your "ray_results" folder. One more problem is checkpoint number. You just need to mention the number and not as "checkpoint_40". Hopefully my below example screenshot helps you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flow-project/flow/issues/522?email_source=notifications&email_token=AH6J3WOENNUEJKMHJBV54VTQBEIORA5CNFSM4HMZBNL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2YFYLA#issuecomment-514874412, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6J3WKDRZUKBOQNZA3I4EDQBEIORANCNFSM4HMZBNLQ .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

pnp91 commented 5 years ago

No problem :) All the best

On Thu, 25 Jul 2019 at 10:14, jingyanping notifications@github.com wrote:

OK,thank you very much for your reply!

At 2019-07-25 16:05:24, "niyathipriya" notifications@github.com wrote: Hello Ping,

yes you can do that. You need to change in the scenario which you define with the specify routes, specify connections and specify edges functions. This is well explained further in tutorial on "how do we create our own custom scenarios". Although building a scenario is not a very easy task, I would suggest you to follow the tutorial given in FLOW. They have appreciable description which helps us to build our own tracks. Hope this helps.

On Thu, 25 Jul 2019 at 04:52, jingyanping notifications@github.com wrote:

hello， I'm sorry trouble you again. i have trained the auto car in a closed network using PPO algorithm, it can dissipate stop-and-go waves in the closed loop.(The PPO algorithm can also be replaced with other algorithm,such as A3C,DQN,DDPG,right?). i trained the auto car in closed network, Can it be migrated to an open-loop network,for example, on a single-lane road with one or two ramps? If so,can you give me some advice about how to do that? sincerely jing

At 2019-07-18 16:00:40, "niyathipriya" notifications@github.com wrote:

I think the problem is you used wrong name for .. It is not the name of .py file. But there is a parameter inside .py file called exp_tag, you have to copy that name. It is also the same name which you have inside your "ray_results" folder. One more problem is checkpoint number. You just need to mention the number and not as "checkpoint_40". Hopefully my below example screenshot helps you.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/flow-project/flow/issues/522?email_source=notifications&email_token=AH6J3WOENNUEJKMHJBV54VTQBEIORA5CNFSM4HMZBNL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2YFYLA#issuecomment-514874412 , or mute the thread < https://github.com/notifications/unsubscribe-auth/AH6J3WKDRZUKBOQNZA3I4EDQBEIORANCNFSM4HMZBNLQ

.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flow-project/flow/issues/522?email_source=notifications&email_token=AH6J3WLSBXFWQVKYVUDPL5LQBFOEPA5CNFSM4HMZBNL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2YW7ZA#issuecomment-514944996, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6J3WIWLSRI3D6TFNQGOKTQBFOEPANCNFSM4HMZBNLQ .

eugenevinitsky commented 5 years ago

Hi Jing, this is actually a very good question and something that's not incredibly easy to do at the moment (though we're working on making it easier)! It's definitely possible, but you have to mess around a good deal with rllib to make it work. Basically, you need to figure out how to apply the policy graph that was trained in one environment into another. So, you can do this by instantiating the same "agent" with a new environment. How to do that at the moment is kind of involved. We'll try to make it easier but for the moment I'm sorry to say you may have to do some legwork to figure it out :)

jingyanping commented 5 years ago

Hi eugenevinitsky，I read a paper(Dissipating stop-and-go waves in the closed and open networks via deep reinforcement learning) from your team, which is to transfer the experience gained in the closed network to the open network through transfer learning. But i cant't understand how this is done now. Another problem is that the auto car i trained in the closed network used the PPO algorithm in tutorial 03, i am learning PPO algorithm recently, i want to know if the PPO algorithm code is in ray/python/ray/rllib/agents/ppo/ppo.py? Because i don't understand the training process and reward mechanism of the vehicle, i want to thoroughly understand how the vehicle is trained, just observe the speed and headway of the front car?

jingyanping commented 4 years ago

Dear Niyathipriya，

Hello，I am so sorry to disturb you, I have encountered a problem and I would like to ask you. In the experiment of using the ‘AV’ car to dissipate the loop stop-and-go wave, in ‘stabilizing_the_ring.py’, when reading the relevant paper, the MLP strategy is selected for the agent. but I did not find the MLP strategy related program in the provided program, but in 'ray/ Python/ray/rllib/models' found programs like fcnet.py, so I was confused. Mentioned in the paper 'Flow:Architecture and Benchmarking for Reinforcement Learning in Traffic Control': "The most experiments,a diagonal Gaussian MLP policy is used with hidden layers (100,50,25) and tanh nonlinearity,The expeiment stabilizing the ring,uses a hidden layer of shape(3,3)..." sincerely Jing

在 2019-07-18 16:35:05，"niyathipriya" notifications@github.com 写道： Cool!

Kindly contact me in case of any concerns.

Thanks & Regards, Niyathipriya Pasupuleti Tel : +4915143598348

On Thu, 18 Jul 2019 at 10:32, jingyanping notifications@github.com wrote:

oh yes, i resolve that problem follow your advice,Thank you very much!!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/flow-project/flow/issues/522?email_source=notifications&email_token=AH6J3WKC6FRFPFEEBR7I6S3QAATB5A5CNFSM4HMZBNL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2HXZNA#issuecomment-512720052, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6J3WL7DCXPY6XQXS6LHT3QAATB5ANCNFSM4HMZBNLQ .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.