search

rll-research / sim2seg

Implementation of Sim2Seg (John So*, Amber Xie*, Sunggoo Jung, Jeffrey Edlund, Rohan Thakker, Ali-akbar Agha-mohammad, Pieter Abbeel, Stephen James)
MIT License
30 stars 6 forks source link

Release Files can't be used #1

Closed hellc3c closed 1 year ago

hellc3c commented 1 year ago
  1. the link of "Sim2Seg segmentation model" is the same as the link of the environment resources file, and I can't find any nn model-like file in that folder.
  2. the environment resources files have different names from your code in buggy.yaml. I have never used ml-agents library to train RL agents, so I can't configure that yaml file straight. please give a more detailed variables replacement annotation, that would be thankful.
hellc3c commented 1 year ago

I find out how to run the sim and finally run the train.py successfully with the use_s2s=False option. But the first problem remains. I assume you are willing to release the pre-trained sim2seg model, In that case, please let me know once you put the right link on the readme.md.

typoverflow commented 1 year ago

Also having trouble running the code here. The pre-trained segmentation model is nowhere to be found, so I just set use_s2s=False for debugging. I can't figure out how to launch the BuggyGCEnv either -- it says missing executable files (see the following error logs). However, the released files in google drive cannot match the required file path. Any instructions on this? Thanks for you time.

image
amberxie88 commented 1 year ago

@typoverflow please change self.BASE_FOLDER here! https://github.com/rll-research/sim2seg/blob/master/train.py#L72

I'll try and upload a s2s model to the drive link in the coming weeks, sorry for the delay! The segmentation model will be one trained on our Unity environments, so no guarantee it will work perfectly for this simple environment. For now, you should be able to run with use_s2s=False per @hellc3c

If you would like to train a segmentation model on our provided Unity environment, you can collect paired randomized data and segmentation masks through a separate script. You would need to modify the Unity environment itself (C++ code, need to install Unity, etc.) and run a data collection-specific script. Once you have generated the dataset, you can train a segmentation model by making modifications to https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

Feel free to let me know if there are any other concerns with this codebase or ml-agents!

amberxie88 commented 1 year ago

Uploaded the model and a corresponding tutorial script: https://github.com/rll-research/sim2seg/blob/master/sim2seg_segmentation.ipynb

Please raise new issues if you come across anything else. Thanks!