Reoganize parts folder and refactor large files

[Ezward]

The parts folder is one the most important sets of source in the donkey project, but it is not well organized. I am proposing two steps to make the parts folder more understandable and maintainable;

1. Move parts into sub-folders organized around robotics functions
2. Refactor large dumping ground files into smaller single-purpose files.
   • It is likely that after a refactor we will find a lot of code that really is not a 'part', in which case we would move that out of the parts folder. For instance, we may want a machine-learning folder outside of the parts folder that deals with models and learning the models. That kind of functionality should not in the 'parts' folder; probably on the inference call needs to be in the parts folder.

Here is an initial proposal. This is a simple first cut; we could go deeper into each file and confirm it's destination. Further, we could look at each file and decide how to refactor it and where those resulting files should live.

Folder are in [square-brackets]. In some cases, I've added forward-looking folders for parts that I've heard folks are working on, like obstacle detection and path planning.

• Organize parts folder under broad robotics functions and refactor large files (or dumping ground files):

  • [perception]

  • [tachometry]

    • encoder.py
    • pigpio_enc.py

  • [imu]

    • imu.py

  • [camera]

    • camera.py // TODO: refactor each camera into it's own file
    • realsense2.py
    • realsense435i.py

  • [proximity]

    • lidar.py

  • [obstacle]

  • [machine_learning] // TODO: probably move most of machine_learning out of parts - leave only the actual parts that do inference in the parts folder

  • [pytorch]

    • fastai.py

  • [tensorflow]

    • keras.py // TODO: refactor into smaller files
    • tflite.py
    • coral.py

  • [tensorrt]

    • tensorrt.py

  • [augmentation]

    • augment.py // TODO: refactor individual augmentations into their own files

  • behavoir.py

  • salient.py

  • [computer_vision]

  • image.py // TODO: refactor into separate files - this is a dumping ground for loosely related parts

  • cv.py // TODO: refactor into separate files inside a cv2 folder - some methods are not parts and should be moved out to general computer_vision folder

  • [localization]

  • [map]

  • [particle_filter]

  • [kalman_filter]

  • [planning]

  • [path]

  • [avoidance]

  • [control]

  • [actuation]

    • actuator.py. // TODO: break out individual actuators into their own files
    • launch.py
    • serialcontroller.py

  • [gamepad]

    • controller.py // TODO: break into base and separate files for each kind of joystick and separate file for joystick creator.

  • [voice_control]

  • [web_controller]

  • robohat.py // TODO: refactor controller and driver into their own files

  • sombrero.py

  • teensy.py

  • throttle_filter.py

  • transform.py // TODO: this has several unrelated classes/methods - refractor into their own files

  • [simulation]

  • dgym.py

  • simulation.py

  • [telemetry]

  • datastore.py

  • file_watcher.py

  • graph.py

  • led_status.py

  • network.py // TODO: network pub/sub messaging: not sure if this is telemetry or related to control (has to do with gym remote control)

  • oled.py

  • ros.py

[sctse999]

I like the idea of having a set of agreed development guidelines so we can keep things organized and easy to follow (so more ppl can contribute easier). I think having sub-folders is a good idea because we might have similar stuff that we want to group together. Test case is something that I concern most. Maybe we want to apply something similar to the test folder.

[Ezward]

I agree about the test folder; it should reflect the structure of the src folder. I will open another issue after we finalize the structure of the src folder.

[wallarug]

I have some views on the structure. It has been improved before.

I like this new structure but need to make a suggestion on the hardware / controller folder. It in my view has always been confusing having actuation and control types for the parts and then custom boards separate. As suggested in the post, they should be refactored into separate parts and stored in the appropriate directory.

You should not have hardware output controllers (PCA9685, Robo HAT MM1, etc) in the same folder as the remote controls. It doesn't make sense and is confusing from a naming convention point of view.

The other suggestion I would like to propose is less directories than that which has been proposed. Everything should be able to fall into one of five top level categories (instead of 8 categories), as below to avoid having lots of things at the top level (as is the case with parts/ at the moment).

--

• [actuation / hardware / outputs]
  • [boards]
    • pca9685.py
    • sombreo.py
    • robohat.py
  • [other]
    • serial_controller.py
    • each individual actuator
  • [extra]
  • throttle_filter.py
  • transform.py

--

• [compute]
  • [tensorflow]
  • [pytorch]
  • [tensorrt]
  • [computer_vision]
  • [localization]
  • [augmentation]

--

• [control / inputs]
  • [web_controller]
  • [boards]
  • robohat.py
  • arduino.py
  • [simulation]
  • [gamepad]
  • [voice_control]

--

• [sensors / sensory]
  • [vision]
  • camera.py
  • realsense.py
  • [proximity]
  • lidar.py
  • hcsr04.py
  • [motion]
  • imu.py
  • encoder.py

-- - [telemetry / communication]

I like where this is heading in general and open to see everyone else's suggestions.