search

Atcold / pytorch-PPUU

Code for Prediction and Planning Under Uncertainty (PPUU)
https://youtu.be/X2s7gy3wIYw
MIT License
203 stars 54 forks source link

eval_policy - episodes id's are different than the past results #28

Closed serkanpep closed 5 years ago

serkanpep commented 5 years ago

It seems that the episode ID's that eval_policy.py currently uses are not the same as the ones that we have in the past results. That is after adjusting the change in directory structure that was made in the commit 6860104. Before that, the episodes were being saved under the folder ep{j} (which is the convention used in the past simulation results that we have for the failure cases). After this commit the simulations are saved under the folder ep{j+1}. So I am making this adjustment when I compare my current results with the past results.

I have confirmed that my data_splits.pth file is the same as the on in the traffic-data-atcold/data_i80_v0 folder.

For visual comparison I have uploaded the simulation videos to the drive folder below https://drive.google.com/drive/u/1/folders/15jH0rCAMGELvoo6FBUu_Tn0kNUo3RTXs The files that have the suffix 'policy_82' are the episodes that were previously saved. When I check the results of the same episodes using the latest code, I am getting the episodes that I have saved with the suffix 'policy_87'.

skarakulak commented 5 years ago

This issue seems to be caused by the glob library. Training, validation and test indices that we have in split.pth are the same since we fix the seed there. But the cars that we select are chosen from dataloader.ids as below: car_path = dataloader.ids[splits['test_indx'][j]] where ids are populated like this: ids = glob.glob(f'{data_dir}/{df}/car*.pkl') As a result, the car that we end up selecting depends upon the order of the elements that we have in dataloader.ids. However the order of elements that glob() returns depends upon the order of elements in the filesystem. So they are arbitrary and the result will change as soon as we start working in a different directory.

Atcold commented 5 years ago

Yes, I was aware of the non sorted output of glob.

ipdb> ids = self.ids                                                
ipdb> ids_sorted = ids.copy()                                       
ipdb> len(ids)                                                      
5596
ipdb> ids[:3]                                                       
['./traffic-data/state-action-cost/data_i80_v0/trajectories-0500-0515/car285.pkl',
'./traffic-data/state-action-cost/data_i80_v0/trajectories-0500-0515/car290.pkl',
'./traffic-data/state-action-cost/data_i80_v0/trajectories-0500-0515/car286.pkl']
ipdb> ids_sorted.sort()                                             
ipdb> ids_sorted[:3]                                                
['./traffic-data/state-action-cost/data_i80_v0/trajectories-0400-0415/car1.pkl',
'./traffic-data/state-action-cost/data_i80_v0/trajectories-0400-0415/car100.pkl',
'./traffic-data/state-action-cost/data_i80_v0/trajectories-0400-0415/car1001.pkl']