Open tinkei opened 1 year ago
Hey @tinkei, thanks for reporting. The issue isn't fully resolved, but I'll make some changes to make it more robust. Two questions:
1) to sanity-check, are you using the pre-cached features? you should have arguments like
cache.cache_path={CACHE_PATH} cache.use_cache_without_dataset=True
2) I think what's happening is a feature cache file is being created, but doesn't get fully written to before training finishes. Could fix this by writing to a temp file, then moving the result once we compute and write the feature. Can you confirm this is the case by checking if an empty file exists for the feature that fails?
I'll also try to replicate. Thanks!
Hello, is there an update on this? The same problem happens if you interrupt caching and rerun the caching with same cache directory.
Describe the bug
Training interrupted by error
Failed to compute features for scenario token XXX in log XXX
Error: Ran out of input
. Looking at closed issues, I thought it's already fixed long back in v0.4.Setup
Please share your setup with us, the more detail the better. For example, type of machine (laptop, cluster instance), linux distribution, no. of cpu, no. of gpus, RAM, VRAM, cuda version, conda environment, nuplan-devkit release version.
Steps To Reproduce
Steps to reproduce the behavior:
python nuplan/planning/script/run_training.py ...
python nuplan/planning/script/run_training.py ...
again, specifying the same cache directoryStack Trace
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
It appears that this error arise after I keyboard-interrupt (Ctrl+C) a previous run of
python nuplan/planning/script/run_training.py ...
, when I attempt to run a new experiment that shares the same cache directory.