Fixing worker random seeds and cleaning up debugging

BerkeleyAutomation / gqcnn

Python module for GQ-CNN training and deployment with ROS integration.

https://berkeleyautomation.github.io/gqcnn

Other

312 stars 149 forks source link

Fixing worker random seeds and cleaning up debugging #66

Closed visatish closed 5 years ago

visatish commented 5 years ago

Fixing worker random seeds:

Worker processes are now explicitly passed a unique random seed to prevent them from all using the same one as the parent process.

Cleaning up debugging:

Debug flag in training config is now actually used.
Debug flag and random seed for gqcnn config are now linked to same parameters in training config.
Scripts train.py and finetune.py no longer set None random seeds.

Misc:

No longer attempt to close Tensorboard process when it hadn't actually started (can happen when an exception is raised earlier on in the training pipeline).

visatish commented 5 years ago

@jeffmahler There was a bit of a race condition here where I updated this PR right when you merged it. I just cleaned up some things for debugging, ex. we weren't actually properly using the debug config parameter. Do you mind taking another glance at the PR? If there are any issues we can revert.

jeffmahler commented 5 years ago

@visatish Are you sure those weren't accepted in the PR? I saw the config file changes for debugging when I merged.

To be more specific, is there a particular commit that wasn't merged?

visatish commented 5 years ago

@jeffmahler Oh no, it all got merged-I just thought you might not have gotten a chance to look over the more recently pushed commits.

jeffmahler commented 5 years ago

@visatish The extra commits looked ok to me but I admit I didn't test them

visatish commented 5 years ago

@jeffmahler I tested them out a fair amount so hopefully everything's okay haha. Just realized that there's an extraneous print statement here, but I can fix that directly in master.