Phase 2 deliverable: removal of flight stack

[aditya00j]

All flight stack related applications, support libraries, and irrelevant testing, post-processing, and example scripts need to be removed. This issue can only be closed after we have successfully built the code after these modifications.

[aditya00j]

As I had said during our initial discussions, what we are doing is very similar to what is being done by NavStik for their PandaPilot software. It is also a fork of PX4, but re-packaged for their own flight stack. You can check it out here: https://github.com/navstik/pandapilot_v4 It might be a good reference to follow.

[priyablue]

Yes we will do.

On 03-Nov-2017 12:39 PM, "aditya00j" notifications@github.com wrote:

As I had said during our initial discussions, what we are doing is very similar to what is being done by NavStik for their PandaPilot software. It is also a fork of PX4, but re-packaged for their own flight stack. You can check it out here: https://github.com/navstik/pandapilot_v4 It might be a good reference to follow.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/aditya00j/GX/issues/7#issuecomment-341634238, or mute the thread https://github.com/notifications/unsubscribe-auth/AMbVv9ni3Q2zNGx3bVFpOILxftYmUPh4ks5syrwsgaJpZM4QMNRB .

[aditya00j]

In this phase, you will have to remove files and folders mainly from src.

Please start by identifying what each module does in src/modules. Many of them (e.g. fw_att_control, ekf2 etc.) will be about flight stack. However, some of them (e.g. uorb, sdlog2) etc. will be about hardware abstraction. We want to retain the hardware abstraction layer and remove the flight stack.

You should also be able to remove a large amount of libraries like src/lib/controllib, mathlib, matrix, as long as they are not being used by the retained modules. Please do a careful dependency analysis of the modules to make sure that we remove absolutely everything that is not being used.

[priyablue]

You are right!, but all these will be removed in Phase 2. Once you get the successful build for phase1 we will upload the phase 2. SO we are waiting for your confirmation on successful build.

[priyablue]

In this phase do you need to do anything with debug & integration folder?

[priyablue]

Sorry its debug & integrationtests folder.

[aditya00j]

We can clean that up in Phase 3. Phase 2 and Phase 3 should not be too different, in Phase 3 we will remove other nonessential stuff - debig, integrationtests, test_data, and even a lot of things in Tools can probably go. Also, I don't completely understand ROS integration, but a lot of the ros launch scripts in launch folder also look irrelevant.

[priyablue]

Ok got it. Since we are going to go with our custom application stack, we can remove ROS launch files too as they are more concerned with examples and not with our custom application . So as you mentioned , we can remove all launch files and retain only the support of ROS so that the users/ your team can write separate launch for ROS when you are working with it

[aditya00j]

Hi, I am copying nuttx_px4fmu-v3_default.cmake below, and the modules that are not needed are marked in bold and with "$$$" at the beginning of the line. For most of them, I've written a comment about why it is not needed. However, I don't know fully well the module dependencies (especially about the library modules), so if any of them are required by other modules, we will have to revisit this list.

include(nuttx/px4_impl_nuttx) px4_nuttx_configure(HWCLASS m4 CONFIG nsh ROMFS y ROMFSROOT px4fmu_common IO)

set(config_uavcan_num_ifaces 2)

set(config_module_list #

Board support modules

# drivers/airspeed drivers/blinkm drivers/bmi160 drivers/bmp280 drivers/boards drivers/bst drivers/camera_trigger drivers/device drivers/ets_airspeed drivers/frsky_telemetry drivers/gps drivers/hmc5883 drivers/hott drivers/hott/hott_sensors drivers/hott/hott_telemetry drivers/iridiumsbd drivers/l3gd20 drivers/led drivers/lis3mdl drivers/ll40ls drivers/lsm303d drivers/mb12xx drivers/mkblctrl drivers/mpu6000 drivers/mpu9250 drivers/ms4525_airspeed drivers/ms5525_airspeed drivers/ms5611 drivers/oreoled drivers/pwm_input drivers/pwm_out_sim drivers/px4flow drivers/px4fmu drivers/px4io drivers/rgbled drivers/sdp3x_airspeed drivers/sf0x drivers/sf1xx drivers/srf02 drivers/stm32 drivers/stm32/adc drivers/stm32/tone_alarm drivers/tap_esc drivers/teraranger drivers/vmount modules/sensors

#

System commands

# systemcmds/bl_update systemcmds/config systemcmds/dumpfile systemcmds/esc_calib systemcmds/hardfault_log systemcmds/led_control systemcmds/mixer systemcmds/motor_ramp systemcmds/motor_test systemcmds/mtd systemcmds/nshterm systemcmds/param systemcmds/perf systemcmds/pwm systemcmds/reboot systemcmds/sd_bench systemcmds/top systemcmds/topic_listener systemcmds/ver

#

Testing

# drivers/sf0x/sf0x_tests drivers/test_ppm

lib/rc/rc_tests

$$$ modules/commander/commander_tests : not needed since commander is a flight stack app $$$ lib/controllib/controllib_test: controllib is not needed modules/mavlink/mavlink_tests $$$ modules/mc_pos_control/mc_pos_control_tests: flight stack app modules/uORB/uORB_tests systemcmds/tests

#

General system control

# $$$ modules/commander: flight stack app modules/events modules/gpio_led $$$ modules/land_detector: flight stack app modules/load_mon modules/mavlink $$$ modules/navigator: flight stack app modules/uavcan modules/camera_feedback

#

Estimation modules

# $$$ modules/attitude_estimator_q: flight stack app $$$ modules/ekf2: flight stack app $$$ modules/local_position_estimator: flight stack app $$$ modules/position_estimator_inav: flight stack app

#

Vehicle Control

# $$$ modules/fw_att_control: flight stack app $$$ modules/fw_pos_control_l1: flight stack app $$$ modules/gnd_att_control: flight stack app $$$ modules/gnd_pos_control: flight stack app $$$ modules/mc_att_control: flight stack app $$$ modules/mc_pos_control: flight stack app $$$ modules/vtol_att_control: flight stack app

#

Logging

# modules/logger modules/sdlog2

#

Library modules

# modules/dataman modules/systemlib/param modules/systemlib modules/systemlib/mixer modules/uORB

#

Libraries

# $$$ lib/controllib: will be required only by flight stack apps $$$ lib/conversion: will be required only by flight stack apps lib/DriverFramework/framework $$$ lib/ecl: will be required only by flight stack apps $$$ lib/geo: will be required only by flight stack apps $$$ lib/geo_lookup: will be required only by flight stack apps $$$ lib/launchdetection: will be required only by flight stack apps lib/led $$$ lib/mathlib: will be required only by flight stack apps $$$ lib/mathlib/math/filter: will be required only by flight stack apps $$$ lib/runway_takeoff: will be required only by flight stack apps $$$ lib/tailsitter_recovery: will be required only by flight stack apps $$$ lib/terrain_estimation: will be required only by flight stack apps lib/version

#

Platform

# platforms/common platforms/nuttx platforms/nuttx/px4_layer

#

OBC challenge

# $$$ examples/bottle_drop: irrelevant

#

Rover apps

# $$$ examples/rover_steering_control: irrelevant

#

Segway

# $$$ examples/segway: irrelevant

#

Demo apps

#

Tutorial code from

https://px4.io/dev/px4_simple_app

examples/px4_simple_app

Tutorial code from

https://px4.io/dev/daemon

examples/px4_daemon_app

Tutorial code from

https://px4.io/dev/debug_values

examples/px4_mavlink_debug

Tutorial code from

https://px4.io/dev/example_fixedwing_control

$$$ examples/fixedwing_control: not required

Hardware test

examples/hwtest

EKF

$$$ examples/ekf_att_pos_estimator: not required )

[priyablue]

There are some problems prevailing in the controllib packages removal. For example, the "Battery.cpp" in the GX/src/modules/systemlib folder has the header defined with superblock constructor which is dependant on blocks.hpp (this calls all files in controllib) in controllib folder. So if need to remove then have to rewrite the entire battery file. So is it recommended to do that ? if so do let us know will proceed in the way to eliminate the mentioned files.

[priyablue]

Have uploaded the phase2 build under the new branch "phase2" , with the possible removal of the estimation and application packages. Will proceed from this as per your advice.

[aditya00j]

Ok, let me check the controllib package. I'll get back to you about it by the end of the day. Meanwhile, you can retain that package and proceed with the rest.

[priyablue]

So can we proceed for next phase then ?

[priyablue]

Kindly check the phase2 build and let me know.

[aditya00j]

Are all the other flight stack packages apart from controllib removed already? I'll check the Phase 2 build, please give me a couple of hours.

[priyablue]

In testing part of the px4fmu default cmake file, either we should rewrite the file or we should include it.

If we remove "modules/commander/commander_tests " this is the error we get. src/systemcmds/tests/libsystemcmds__tests.a(tests_main.c.obj):(.rodata.tests+0x88): undefined reference to `commander_tests_main'

If we remove " modules/mc_pos_control/mc_pos_control_tests" this is the error we get NuttX/apps/libapps.a(builtin_list.o):(.rodata.g_builtins+0x29c): undefined reference to `mc_pos_control_tests_main'

The same kind of error we get in the some of the above mentioned files in Testing, general control systems, estimation modules and libraries path.

Rest other than the list given above is already removed in phase2

Ok let me know what can we do to sort out this issue. Advise is needed on this

[aditya00j]

It appears that src>systemcmds>tests>tests_main.h includes the tests for many of the modules that we want to remove (such as commander and mc_pos_control). PX4 is treating these modules as "system" modules even though they represent the flight stack. So we will need to remove these modules from tests_main.

This is a flaw in PX4 software design, where they haven't properly separated hardware abstraction from flight stack. Unfortunately, we have to work to separate these two.

[priyablue]

Thanks for the advice. we will look into and take care of it and update you by end of the day.

[aditya00j]

While going through the changes between phase1 and phase2 branches, I noticed that you have removed px4esc-v1 and px4flow-v2 from src/drivers/boards. These will be needed, since px4esc supports ESC control and px4flow supports PX4Flow optic flow smart camera. These are peripherals which need to be supported by our hardware.

Please re-verify once.

[aditya00j]

Also, some of the flight stack modules were not included in the cmake file, but they exist in the source code nonetheless. I think they need to go as well. Following is the list of modules that will be related to flight stack (that need to go):

attitude_estimator_q commander ekf2 fw_pos_control_l1 land_detector local_position_estimator mc_pos_control navigator position_estimator_inav

[aditya00j]

Also, have you uploaded the Phase 1 summary of changes document? I couldn't find it either in phase1 or on master branch. Let's make sure the documentation does not fall behind, because as we start modifying the code more and more, it will be important to keep track of modifications and our rationale behind it. In future, it may happen that some of the changes we are making right now will have to be reversed based on application, at that time there should be enough documentation to enable it.

[priyablue]

The revised phase 2 build has been uploaded after doing the following:

1. Irrelevant file linked to Cmake and ROMFS has been removed.
2. Tests files in the /Firmware/src/systemcmds has been modified to run without commander, controllib and mc_pos_control.
3. The contents in the ../src/modules folder cant be removed because many files which are more concerned with the MavLINK and uorb messaging files. Eg: Sensors like barometer, accelerometer, gps are dependant on the mathlib, pos_estimator etc and device files like battery depends upon the controllib package.

So in the ROMFS folder the maximum possible packages which are found irrelevant has been removed in the px4-common and px4-tests folder.

It is recommended to retain the ../src/lib and ../src/modules in phase2 build and this can tested/discussed again for possibilities of removal while running with MATLAB / nsh shell ouput for sensor data logging via MavLINK / Serial Tx/Rx.

you can clone, build and run the build using the following commands:

$ git clone -b phase2 https://github.com/aditya00j/GX.git $ cd $ make list_config_targets $ make

Also upload and test in board.

Let us know for further discussion on this

[aditya00j]

Hi, I understand that some of the common libraries may not be deleted because they are being used by other modules. But what is the problem in deleting the modules like mc_pos_control and fw_pos_control_l1? These are flight stack modules, which we need to remove.

You are right, they will interact with other modules through uORB messages. But that's exactly the thrust of what we want to achieve - e.g. if there is a uORB message being published by mc_pos_control, this message will later be published by our module. As far as I understand, this should not create compilation errors. You should be able to remove these modules. Please check the dependencies carefully, and let's try to remove these modules one by one. For example, you can start by letting me know what the errors are when you try to remove fw_pos_control_l1.

Current Phase2 branch also contains many modules in src>examples folder which we haven't removed yet , e.g. bottle_drop. Are you facing difficulties with these too?

Also, there are some other modules which I had missed earlier, which need to be removed: lib/launchdetection lib/runwaytakeoff lib/tailsitterrecovery lib/terrainestimation lib/conversion I missed them because they were not included in the cmake file which we parsed earlier. I will let you know if I find any more such modules.

[aditya00j]

These are the modules that must be removed from src/examples: attitude_estimator_ekf bottle_drop ekf_att_pos_estimator fixedwing_control mc_att_control_multiplatform mc_pos_control_multiplatform rover_steering_control segway uuv_example_app

Other cleanup (such as test_data folder, scripts in Tools etc.) we will take up in Phase 3. But let's remove these flight stack related modules in this phase.

[bluetulip89]

For example , as you described , if we remove the fw_pos_control_l1 , the file inside this package is required by ../src/modules/navigator/mission_feasibility_checker.cpp . the included library file is given below :

include <fw_pos_control_l1/Landingslope.hpp>

The navigator is inturn called in mavlink mission.cpp as given below: ../src/modules/mavlink/mavlink_mission.cpp

include <navigator/navigation.h>

used in line ----> stats.update_counter = ++_geofence_update_counter;

[aditya00j]

Yes, but the navigator module also needs to be removed. It's possible that these modules are dependent on each other, so we might not be able to remove them one by one. I think the problem will be in mavlink_mission, more than any other places.

Can we set some time apart, may be tomorrow, where we can take a look at this together? I was expecting these dependency issues to come up.

[aditya00j]

I think we need to add back the px4io-v2 in nuttx-configs, as this is needed for Pixhawk 2.1

[priyablue]

Ok will add to it.

[priyablue]

As we discussed , i attached the document regarding the details of packages and libs in ../src/modules and ../src/lib folder.

Src_packages_details.docx

Go through the document and mark your advice in the comments column and resend it within afternoon. @aditya00j

[priyablue]

As you discussed the example files can be removed . It can be done in 3rd phase build

[aditya00j]

Hi, The Phase 2 as it currently stands is passing the build. I suggest that we combine Phase 2 and Phase 3, and work to resolve these issues together. So you need not create a Phase 3 branch. We'll keep working on this branch, and after we close it, we'll go directly to Phase 4 branch.

[aditya00j]

From your document, it appears that following files can be removed right away:

land_detector gnd_att_control gnd_pos_control vtol_att_control geo_lookup launchdetection runway_takeoff tailsitter_recovery terrain_estimation

Please work towards removing them, while I take a look at other modules like Commander, Navigator etc.

[aditya00j]

As we are merging Phase 2 and Phase 3, please also clean up the following folders (as long as there are no unintended consequences):

integrationtests launch test_data Tools/astyle: this seems to check for code style. Not needed. Tools/dist Tools/ecl_ekf Tools/Matlab Tools/models Tools/jMAVSim: this is a git submodule Tools/sitl_gazebo: this is a git submodule Tools/stack_usage: check this module, I am not exactly sure what it does and whether it is needed.

Please also take a look at the individual files in the Tools folder, and remove the irrelevant ones.

[priyablue]

The final build by merge phase2 & phase 3 is been uploaded in phase2 branch as per your instructions. Kindly checkout for any queries / errors and revert back to us if your team have any .

[priyablue]

The following packages is not available for removal as they are directly link with one / more files in systemlib and mavlink . SO they are retained.

1.Geolookup. 2.Launchdetection 3.Runway_takeoff

all the other files as instructed by you and files which we found irrelevant to the build are removed.

the posix-configs and nuttx-configs has been moved to configs folder .

the nuttx folder cant be moved out of platforms folder .

Check whether the folder are arranged as per the directory structure as you discussed in the following issue.

https://github.com/aditya00j/GX/issues/13

[aditya00j]

Hi, I am copying my error stack below. The error seems to be related to jMAVSim path not found. You asked me to ignore this error, but its a fatal error - I can't move ahead.

AJ:GX aditya$ make px4fmu-v3_default
fatal: no submodule mapping found in .gitmodules for path 'Tools/jMAVSim'
-- Build Type: MinSizeRel
Traceback (most recent call last):
  File "Tools/tag_to_version.py", line 19, in <module>
    minor = res[1]
IndexError: list index out of range
-- PX4 VERSION: d6a5d83a8
-- CONFIG: nuttx_px4fmu-v3_default
CMake Deprecation Warning at /Users/aditya/anaconda/share/cmake-3.6/Modules/CMakeForceCompiler.cmake:79 (message):
  The CMAKE_FORCE_C_COMPILER macro is deprecated.  Instead just set
  CMAKE_C_COMPILER and allow CMake to identify the compiler.
Call Stack (most recent call first):
  cmake/toolchains/Toolchain-arm-none-eabi.cmake:31 (cmake_force_c_compiler)
  /Users/aditya/anaconda/share/cmake-3.6/Modules/CMakeDetermineSystem.cmake:98 (include)
  CMakeLists.txt:262 (project)

CMake Deprecation Warning at /Users/aditya/anaconda/share/cmake-3.6/Modules/CMakeForceCompiler.cmake:93 (message):
  The CMAKE_FORCE_CXX_COMPILER macro is deprecated.  Instead just set
  CMAKE_CXX_COMPILER and allow CMake to identify the compiler.
Call Stack (most recent call first):
  cmake/toolchains/Toolchain-arm-none-eabi.cmake:37 (cmake_force_cxx_compiler)
  /Users/aditya/anaconda/share/cmake-3.6/Modules/CMakeDetermineSystem.cmake:98 (include)
  CMakeLists.txt:262 (project)

CMake Deprecation Warning at /Users/aditya/anaconda/share/cmake-3.6/Modules/CMakeForceCompiler.cmake:79 (message):
  The CMAKE_FORCE_C_COMPILER macro is deprecated.  Instead just set
  CMAKE_C_COMPILER and allow CMake to identify the compiler.
Call Stack (most recent call first):
  cmake/toolchains/Toolchain-arm-none-eabi.cmake:31 (cmake_force_c_compiler)
  build/px4fmu-v3_default/CMakeFiles/3.6.3/CMakeSystem.cmake:6 (include)
  CMakeLists.txt:262 (project)

CMake Deprecation Warning at /Users/aditya/anaconda/share/cmake-3.6/Modules/CMakeForceCompiler.cmake:93 (message):
  The CMAKE_FORCE_CXX_COMPILER macro is deprecated.  Instead just set
  CMAKE_CXX_COMPILER and allow CMake to identify the compiler.
Call Stack (most recent call first):
  cmake/toolchains/Toolchain-arm-none-eabi.cmake:37 (cmake_force_cxx_compiler)
  build/px4fmu-v3_default/CMakeFiles/3.6.3/CMakeSystem.cmake:6 (include)
  CMakeLists.txt:262 (project)

-- The ASM compiler identification is GNU
-- Found assembler: /usr/local/bin/arm-none-eabi-gcc
-- Found PythonInterp: /Users/aditya/anaconda/bin/python (found version "2.7.13") 
-- Found PY_jinja2: /Users/aditya/anaconda/lib/python2.7/site-packages/jinja2  
-- C compiler: arm-none-eabi-gcc (GNU Tools for ARM Embedded Processors) 5.4.1 20160919 (release) [ARM/embedded-5-branch revision 240496]
-- C++ compiler: arm-none-eabi-g++ (GNU Tools for ARM Embedded Processors) 5.4.1 20160919 (release) [ARM/embedded-5-branch revision 240496]
-- Building and including 
CMake Error at src/modules/uavcan/CMakeLists.txt:52 (add_subdirectory):
  The source directory

    /Users/aditya/Documents/GX/src/modules/uavcan/libuavcan

  does not contain a CMakeLists.txt file.

CMake Error at src/modules/uavcan/CMakeLists.txt:53 (add_dependencies):
  Cannot add target-level dependencies to non-existent target "uavcan".

  The add_dependencies works for top-level logical targets created by the
  add_executable, add_library, or add_custom_target commands.  If you want to
  add file-level dependencies see the DEPENDS option of the add_custom_target
  and add_custom_command commands.

CMake Error at CMakeLists.txt:425 (add_subdirectory):
  add_subdirectory given source "src/lib/DriverFramework/framework" which is
  not an existing directory.

CMake Error at CMakeLists.txt:425 (add_subdirectory):
  The source directory

    /Users/aditya/Documents/GX/src/lib/ecl

  does not contain a CMakeLists.txt file.

-- NuttX: px4fmu-v3 nsh cortex-m4
-- ROMFS: px4fmu_common
-- Configuring incomplete, errors occurred!
See also "/Users/aditya/Documents/GX/build/px4fmu-v3_default/CMakeFiles/CMakeOutput.log".
/bin/sh: line 0: cd: /Users/aditya/Documents/GX/build/px4fmu-v3_default: No such file or directory
make: *** [px4fmu-v3_default] Error 1

[bluetulip89]

Do give submodule update and try it again ... here we are not facing any errors.

The Path for the jmavsim has been removed from .gitmodules and so this is showing as fatal error as this is been called into main files. You can ignore this as this will pass the build. ( we will remove these non-used links like jMAVsim and sitl-gazebo later)

for the uavcan , driverframework subdirectory paths ... you can make a submodule sync and submodule update then proceed to avoid the errors

[aditya00j]

Same error:

AJ:GX aditya$ git submodule update --recursive --init
fatal: No url found for submodule path 'Tools/jMAVSim' in .gitmodules

[aditya00j]

You can try cloning a new repo from GitHub again and running. Maybe you are not getting the error because your copy already contains the proper submodules.

[bluetulip89]

After this is not proceeding ?

If that so for temporary build , we will include the path and later after getting all done ... will remove all submodules.

now will re upload with modified gitmodule

[bluetulip89]

for the matlab , kindly type "mex -setup" in >>> shell . give the version and name of compiler so that it will be easier for us to use the same here for compilation

[aditya00j]

Our Matlab is set up in Windows, and using MS Visual C++ 2017 compiler. But mex compilation shouldn't be too big a problem, can you share the source code that you are trying to compile? I'll try compiling it in our machine.

[bluetulip89]

We are using your source code from github account only. When we run it matlab it gives out the error as backward compatibility not possible and mex file is invalid . So we asked your compiler version so that we will try using the same with the 2016b version we have

will let you know once we finish this and also will upload the code with the modification in submodule files as you instructed . (post the instructions here , the one you discussed with priya regarding the removal of submodules , for reference)

[aditya00j]

Ok, if you are running my code from here (https://github.com/aditya00j/simulink_mavlink), don't use the compiled mex functions. Try to follow the example given, and generate your own source files and compile them. It's explained in readme, let me know if you have any issues. If you are not able to open the "example_use_sfunctions.slx" file, try this: open Simulink Preferences > Model File and uncheck "do not load models created with a newer version of Simulink"

[bluetulip89]

ok will checkout.

[aditya00j]

Check the document attached. I will discuss on the phone.

GX uORB Messages and Modules.docx

[priyablue]

Kindly go through the document , we have modified all the uORB message that you have marked with "N". All these uORB message has been taken from "px4_stable" uploaded in new branch. You can ignore which ever files that is not required. We will start the re-base work and upload in the new branch.

And let me know if you have any clarifications. GX.uORB.Messages.and.Modules.docx

[priyablue]

AS informed i have modified the paths "Y" also. Kindly check out. GX.uORB.Messages.and.Modules.docx

[aditya00j]

Ok, thanks. I will work on this tomorrow.