About collecting data

[Lesliewsq]

since official stable leaderboard have memory leak issue. Please check your memory and kill it to resume. When I run the data-collecting script，I meet the memory leak problem. After the script for collecting data ran for more than three hours, the program terminated and reported an error numpy.core._exceptions._ArrayMemoryError: Unable to allocate 170. MiB for an array with shape (178773021,) and data type uint8. Can I continue running the program to collect data from the checkpoint, or do I have to start the collection from scratch (meaning the previous data is invalid).

[Kin-Zhang]

collect data from the checkpoint, you can do this.

Set the resume=True it will auto-resume from the checkpoint.

[Lesliewsq]

Thanks for your reply. When I start the collection from the checkpoint, I get an error like this at the beginning. This error seems to make my memory usage increase faster.

 G:\mmfn-main>python run_steps/phase0_run_eval.py port=2000 towns="['Town01', 'Town03', 'Town06', 'Town07']" resume=True if_open_carla=True
[2023-09-30 13:40:25,250][utils][INFO] - D:\pythonProject\WindowsNoEditor\CarlaUE4.exe -quality-level=Epic -carla-rpc-port=2000
 =======> input is a folder, start with for loop
--------------------TEST Agent: mmfn_pilot.py--------------------
Contiune the route from file: G:\mmfn-main\data\results\routes_town01_short_mmfn_pilot.json
> Registering the global statistics
Contiune the route from file: G:\mmfn-main\data\results\routes_town01_tiny_mmfn_pilot.json
Exception ignored in: <function LeaderboardEvaluator.__del__ at 0x000002C9B09643A8>
Traceback (most recent call last):
  File "G:\mmfn-main\leaderboard\leaderboard\leaderboard_evaluator.py", line 118, in __del__
    self._cleanup()
  File "G:\mmfn-main\leaderboard\leaderboard\leaderboard_evaluator.py", line 130, in _cleanup
    if self.manager and self.manager.get_running_status() \
AttributeError: 'LeaderboardEvaluator' object has no attribute 'manager'

I also found that in some route scenarios, the following error will occur.

========= Preparing RouteScenario_111 (repetition 0) =========
> Setting up the agent
Data save path G:\mmfn-main\data\nss_0702\Town01_tiny\Town01_111_09_30_13_50_33
> Loading the world
Skipping scenario 'Scenario4' due to setup error: list index out of range
> Running the route
Loading success for map, save path is: G:\mmfn-main\data\nss_0702\Town01_tiny\Town01_111_09_30_13_50_33
start to build rmap. map_number: 1
Please install the lib through readme, there will not vector map saved!
> Stopping the route

There seems to be a problem with the setup of Scenario4. Is this normal?

[Kin-Zhang]

First one is not normall not sure if it's CARLA 0.10 version. It cannot solve you can try Docker also.

Not normal at least for me, I don't have this issue when I collected, if it's still exist, recommend just delete Scenario4 in the leaderboard/data/all_towns_traffic_scenarios.json

[Lesliewsq]

Thanks for your reply. Still the same problem. The program always stops after collecting for two or three hours, causing me to always need to restart it manually. Is it because I use windows os to run the program? Or ，where can I download this data set?

[Kin-Zhang]

I think it's normally that need resume manually, check this issue for CARLA itself: https://github.com/carla-simulator/leaderboard/issues/81

Maybe I will recommend you collect short and long routes for the dataset. Maybe 100K is enough to train a fine-fine mmfn e2e agent.

Unfortunately, I didn't provide the dataset since the discussion here: https://github.com/Kin-Zhang/mmfn/issues/11#issuecomment-1652296883

[Kin-Zhang]

or now, you can direcly download the weight file and following the README step to test the e2e agent directly. check here: https://github.com/Kin-Zhang/mmfn/issues/14