Currently image augmentations and transformation are a fixed set. There is no way to add a custom augmentation or transformation without changing core code. We can do better.
This was initiated after talking to a user on Discord that wanted to add Canny edge detection as a transformation. They ended up just hacking the camera part to apply the Canny algorithm on the way out; so they end up with transformed data in their data sets, which is not ideal.
Goal:
make it easy to add a custom augmentation or transformation to the training and inference pipelines.
make it easier to maintain the training and inference pipelines; pulling code out of it and into a place where it can be more easily maintained. As an example, we have a nasty bug caused by imgaug on the Jetson that makes it impossible to use cropping; it should be easy to work around this for even fix this.
make code more composable and reusable. We have a lot of parts in cv.py that could be used for augmentation or transformation. For instance, we have a part that will apply Canny edge detection to an image. It takes an image in the run method and returns an image. It was designed as a useful part for the computer vision template. However, that should also be very easy to reuse as transformation.
Make it easier for community to contribute. Regarding hacking the camera part to apply the Canny algorithm; there is no way we would accept a PR that did that, so we cannot leverage that user's innovation.
It would be ideal is user's did not have to change their configuration that uses the current standard set of augmentations and transformations.
This could make it easy to do this in a declarative way; like we intend to do for vehicle construction.
Tasks:
refactor the current augmentations and transformations into parts. The constructor will take in whatever configuration is needed. The run() method will take an image and return an image.
implement a way in the pipeline code (augmentations.py) to add a part with a name to the pipeline. So a part would be constructed, then add(name, part) method would be called with a name and the part instance, similar to how we add a part to the vehicle loop. The pipeline would keep a dictionary of named parts that had had been added.
Change how we manage the augmentations and transformations. Current the pipeline uses what is effectively a switch/case statement in donkeycar/pipeline/augmentations.py. Instead we will have a dictionary and we will lookup the part using it's name.
Change how augmentation/transformation is applied. The current augmentations/transformation call aug_img_arr = self.augmentations.augment_image(img_arr); so the augment_image() method takes an image and returns an image. This would be replaced by the part's run(self, image) method which would take an image and return an image.
add code in complete.py that adds the standard augmentations and transformations to the pipeline using the current names as specified in cfg_complete.py. This is how we handle backwards compatibility with folks' current myconfig.py.
I think that is all we need to do. With that in place a user could add code to their manage.py to add a new augmentation or transformation using one of the cv.py parts or a part of their own creation.
Ezward
commented
1 year ago
Currently image augmentations and transformation are a fixed set. There is no way to add a custom augmentation or transformation without changing core code. We can do better.
This was initiated after talking to a user on Discord that wanted to add Canny edge detection as a transformation. They ended up just hacking the camera part to apply the Canny algorithm on the way out; so they end up with transformed data in their data sets, which is not ideal.
Goal:
cv.pythat could be used for augmentation or transformation. For instance, we have a part that will apply Canny edge detection to an image. It takes an image in the run method and returns an image. It was designed as a useful part for the computer vision template. However, that should also be very easy to reuse as transformation.Tasks:
run()method will take an image and return an image.augmentations.py) to add a part with a name to the pipeline. So a part would be constructed, thenadd(name, part)method would be called with a name and the part instance, similar to how we add a part to the vehicle loop. The pipeline would keep a dictionary of named parts that had had been added.donkeycar/pipeline/augmentations.py. Instead we will have a dictionary and we will lookup the part using it's name.aug_img_arr = self.augmentations.augment_image(img_arr); so theaugment_image()method takes an image and returns an image. This would be replaced by the part'srun(self, image)method which would take an image and return an image.complete.pythat adds the standard augmentations and transformations to the pipeline using the current names as specified incfg_complete.py. This is how we handle backwards compatibility with folks' currentmyconfig.py.I think that is all we need to do. With that in place a user could add code to their manage.py to add a new augmentation or transformation using one of the cv.py parts or a part of their own creation.
PS: this is a way we can fix the imgaug issue as well; https://github.com/autorope/donkeycar/issues/970