Move to ninja build system

tannewt commented 4 years ago

Prototype, WIP version of this is here: https://github.com/adafruit/circuitpython/compare/master...tannewt:ninja

omsai commented 4 years ago

For anyone working on this, switch to the ninja git branch, generate and run the build.ninja file using:

$ python3 tools/build/circuitpython.py feather_m0_express
$ ninja -C build-feather_m0_express

At the minute it fails with:

In file included from .././py/lexer.h:32,
                 from ../extmod/vfs.h:29,
                 from ../main.c:30:
.././py/qstr.h:42:10: fatal error: genhdr/qstrdefs.enum.h: No such file or directory
   42 | #include "genhdr/qstrdefs.enum.h"

...probably because it's not yet running the equivalent make command to generate the qstrdefs.enum.h target:

$ make -Bn -C ports/atmel-samd BOARD=feather_m0_express
...
: "GEN build-feather_m0_express/genhdr/qstrdefs.enum.h"
python3 ../../py/makeqstrdata.py build-feather_m0_express/genhdr/qstrdefs.preprocessed.h > build-feather_m0_express/genhdr/qstrdefs.enum.h
...

Note that you can't run the above make command on the ninja branch; you have to run it on the master branch.

omsai commented 4 years ago

Yesterday I read through the short ninja build manual and wonder if it's worth using the depfile feature that automatically discovers required header files by using gcc -MM -MF $out.d ... to write out all required headers for a given C file. Using depfile could help in several ways:

Besides eliminating the need to specify the implicit header dependencies for an given C file using | foo.h, it also makes the process more robust by ensuring no header files fall through. For example, the current generated ninja.build has:

rule compile
  command = arm-none-eabi-gcc -I../. -I../ports/atmel-samd -I../build-feather_m0_express -I../build-feather_m0_express/qstr -I../build-feather_m0_express/version -I../ports/atmel-samd/asf4/samd21 -I../ports/atmel-samd/asf4/samd21/CMSIS/Include -I../ports/atmel-samd/boards/feather_m0_express -DFFCONF_H=\"lib/oofatfs/ffconf.h\" -D__ATSAMD21G18A__ -DSAMD21 -DSPI_FLASH_FILESYSTEM -DEXTERNAL_FLASH_DEVICE_COUNT=2 -DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" -c $in -o $out

build qstr/pp/main.c: qstr_src_pp ../main.c | version/genhdr/mpversion.h

Using depfile this can be changed to:

rule compile
  command = arm-none-eabi-gcc -MM -MF $out.d -I../. -I../ports/atmel-samd -I../build-feather_m0_express -I../build-feather_m0_express/qstr -I../build-feather_m0_express/version -I../ports/atmel-samd/asf4/samd21 -I../ports/atmel-samd/asf4/samd21/CMSIS/Include -I../ports/atmel-samd/boards/feather_m0_express -DFFCONF_H=\"lib/oofatfs/ffconf.h\" -D__ATSAMD21G18A__ -DSAMD21 -DSPI_FLASH_FILESYSTEM -DEXTERNAL_FLASH_DEVICE_COUNT=2 -DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" -c $in -o $out
  depfile = $out.d

build qstr/pp/main.c: qstr_src_pp ../main.c

This is what the gcc generated depfile main.o.d looks like so far. Towards the bottom you can see mpversion.h as ../build-feather_m0_express/version/genhdr/mpversion.h because we're saving main.o under objs/ which modify the relative path:

main.o: ../main.c ../extmod/vfs.h ../py/lexer.h ../py/mpconfig.h \
 ../ports/atmel-samd/mpconfigport.h \
 ../ports/atmel-samd/asf4/samd21/include/sam.h \
 ../ports/atmel-samd/asf4/samd21/include/samd21g18a.h \
 ../ports/atmel-samd/asf4/samd21/CMSIS/Include/core_cm0plus.h \
 ../ports/atmel-samd/asf4/samd21/CMSIS/Include/cmsis_compiler.h \
 ../ports/atmel-samd/asf4/samd21/CMSIS/Include/cmsis_gcc.h \
 ../ports/atmel-samd/asf4/samd21/include/system_samd21.h \
 ../ports/atmel-samd/asf4/samd21/include/component/adc.h \
 ../ports/atmel-samd/asf4/samd21/include/component/dac.h \
 ../ports/atmel-samd/asf4/samd21/include/component/dmac.h \
 ../ports/atmel-samd/asf4/samd21/include/component/dsu.h \
 ../ports/atmel-samd/asf4/samd21/include/component/eic.h \
 ../ports/atmel-samd/asf4/samd21/include/component/evsys.h \
 ../ports/atmel-samd/asf4/samd21/include/component/gclk.h \
 ../ports/atmel-samd/asf4/samd21/include/component/hmatrixb.h \
 ../ports/atmel-samd/asf4/samd21/include/component/i2s.h \
 ../ports/atmel-samd/asf4/samd21/include/component/mtb.h \
 ../ports/atmel-samd/asf4/samd21/include/component/nvmctrl.h \
 ../ports/atmel-samd/asf4/samd21/include/component/pac.h \
 ../ports/atmel-samd/asf4/samd21/include/component/pm.h \
 ../ports/atmel-samd/asf4/samd21/include/component/port.h \
 ../ports/atmel-samd/asf4/samd21/include/component/rtc.h \
 ../ports/atmel-samd/asf4/samd21/include/component/sercom.h \
 ../ports/atmel-samd/asf4/samd21/include/component/sysctrl.h \
 ../ports/atmel-samd/asf4/samd21/include/component/tc.h \
 ../ports/atmel-samd/asf4/samd21/include/component/tcc.h \
 ../ports/atmel-samd/asf4/samd21/include/component/usb.h \
 ../ports/atmel-samd/asf4/samd21/include/component/wdt.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/ac.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/adc.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/dac.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/dmac.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/dsu.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/eic.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/evsys.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/gclk.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sbmatrix.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/i2s.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/mtb.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/nvmctrl.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/pac0.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/pac1.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/pac2.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/pm.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/port.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/rtc.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom0.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom1.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom2.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom3.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom4.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sercom5.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/sysctrl.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tc3.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tc4.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tc5.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tcc0.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tcc1.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/tcc2.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/usb.h \
 ../ports/atmel-samd/asf4/samd21/include/instance/wdt.h \
 ../ports/atmel-samd/asf4/samd21/include/pio/samd21g18a.h \
 ../py/circuitpy_mpconfig.h \
 ../ports/atmel-samd/boards/feather_m0_express/mpconfigboard.h \
 ../supervisor/flash_root_pointers.h \
 ../supervisor/shared/external_flash/external_flash_root_pointers.h \
 ../ports/atmel-samd/peripherals/samd/dma.h ../py/qstr.h ../py/misc.h \
 ../py/mpconfig.h ../build-feather_m0_express/genhdr/qstrdefs.enum.h \
 ../py/reader.h ../py/obj.h ../py/mpprint.h ../py/runtime0.h \
 ../supervisor/shared/translate.h ../py/proto.h ../extmod/vfs_fat.h \
 ../lib/oofatfs/ff.h ../lib/oofatfs/ffconf.h ../extmod/vfs.h \
 ../build-feather_m0_express/version/genhdr/mpversion.h ../py/nlr.h \
 ../py/compile.h ../py/parse.h ../py/emitglue.h ../py/frozenmod.h \
 ../py/mphal.h ../ports/atmel-samd/mphalport.h \
 ../supervisor/shared/tick.h ../extmod/virtpin.h ../py/runtime.h \
 ../py/mpstate.h ../py/mpthread.h ../py/nlr.h ../py/objlist.h \
 ../py/objexcept.h ../py/objtuple.h ../py/pystack.h \
 ../supervisor/linker.h ../py/repl.h ../py/gc.h ../py/stackctrl.h \
 ../lib/mp-readline/readline.h ../lib/utils/pyexec.h \
 ../ports/atmel-samd/background.h ../supervisor/cpu.h \
 ../supervisor/memory.h ../supervisor/port.h \
 ../supervisor/shared/safe_mode.h ../supervisor/filesystem.h \
 ../extmod/vfs_fat.h ../supervisor/shared/autoreload.h \
 ../supervisor/shared/translate.h ../supervisor/shared/rgb_led_status.h \
 ../lib/utils/pyexec.h ../supervisor/port.h \
 ../supervisor/shared/rgb_led_colors.h ../supervisor/shared/safe_mode.h \
 ../supervisor/shared/status_leds.h ../supervisor/shared/stack.h \
 ../supervisor/memory.h ../supervisor/serial.h \
 ../ports/atmel-samd/boards/board.h

omsai commented 4 years ago

I've been experimenting by directly modifying the build.ninja file and can get past the genhdr/qstrdefs.enum.h error, but I don't trust the generated file is corrected because I haven't yet corrected for all the modified paths in the $include_pre, etc. Which is worrying: the ad-hoc sed commands, clearly are not fault tolerant and I think the sed functionality should be merged into the python later. What do you think?

ninja_required_version = 1.1

# These variables are procedurally generated by tools/build/circuitpython.py

includes = $
-I.. $
-I../build-feather_m0_express $
-I../build-feather_m0_express/qstr $
-I../build-feather_m0_express/version $
-I../ports/atmel-samd $
-I../ports/atmel-samd/asf4/samd21 $
-I../ports/atmel-samd/asf4/samd21/CMSIS/Include $
-I../ports/atmel-samd/boards/build-feather_m0_express $
-I../ports/atmel-samd/boards/feather_m0_express

defines = $
-DEXTERNAL_FLASH_DEVICES="S25FL216K,GD25Q16C" $
-DEXTERNAL_FLASH_DEVICE_COUNT=2 $
-DFFCONF_H=\"lib/oofatfs/ffconf.h\" $
-DSAMD21 $
-DSPI_FLASH_FILESYSTEM $
-D__ATSAMD21G18A__

# These variables were reverse engineered from make output and may need to be
# merged with the procedurally generated variables above.

includes_pre = $
-I. $
-I../.. $
-I../../lib/tinyusb/src $
-I../../supervisor/shared/usb $
-I../lib/mp-readline $
-I../lib/timeutils $
-Iasf4/samd21 $
-Iasf4/samd21/CMSIS/Include $
-Iasf4/samd21/hal/include $
-Iasf4/samd21/hal/utils/include $
-Iasf4/samd21/hpl/core $
-Iasf4/samd21/hpl/gclk $
-Iasf4/samd21/hpl/pm $
-Iasf4/samd21/hpl/port $
-Iasf4/samd21/hpl/rtc $
-Iasf4/samd21/hpl/tc $
-Iasf4/samd21/hri $
-Iasf4/samd21/include $
-Iasf4_conf/samd21 $
-Iboards/ $
-Iboards/feather_m0_express $
-Ibuild-feather_m0_express $
-Ifreetouch $
-Iperipherals

defines_pre = $
-DCFG_TUD_CDC_RX_BUFSIZE=128 $
-DCFG_TUD_CDC_TX_BUFSIZE=128 $
-DCFG_TUD_MIDI_RX_BUFSIZE=128 $
-DCFG_TUD_MIDI_TX_BUFSIZE=128 $
-DCFG_TUD_MSC_BUFSIZE=512 $
-DCFG_TUSB_MCU=OPT_MCU_SAMD21 $
-DCIRCUITPY_ANALOGIO=1 $
-DCIRCUITPY_AUDIOBUSIO=1 $
-DCIRCUITPY_AUDIOCORE=1 $
-DCIRCUITPY_AUDIOIO=1 $
-DCIRCUITPY_AUDIOIO_COMPAT=1 $
-DCIRCUITPY_AUDIOPWMIO=0 $
-DCIRCUITPY_BITBANGIO=0 $
-DCIRCUITPY_BITBANG_APA102=0 $
-DCIRCUITPY_BLEIO=0 $
-DCIRCUITPY_BOARD=1 $
-DCIRCUITPY_BUSIO=1 $
-DCIRCUITPY_CANARY_WORD=0xADAF00 $
-DCIRCUITPY_DIGITALIO=1 $
-DCIRCUITPY_DISPLAYIO=1 $
-DCIRCUITPY_FREQUENCYIO=0 $
-DCIRCUITPY_FULL_BUILD=1 $
-DCIRCUITPY_GAMEPAD=1 $
-DCIRCUITPY_GAMEPADSHIFT=0 $
-DCIRCUITPY_I2CSLAVE=0 $
-DCIRCUITPY_MATH=1 $
-DCIRCUITPY_MICROCONTROLLER=1 $
-DCIRCUITPY_NEOPIXEL_WRITE=1 $
-DCIRCUITPY_NETWORK=0 $
-DCIRCUITPY_NVM=1 $
-DCIRCUITPY_OS=1 $
-DCIRCUITPY_PEW=0 $
-DCIRCUITPY_PIXELBUF=1 $
-DCIRCUITPY_PS2IO=0 $
-DCIRCUITPY_PULSEIO=1 $
-DCIRCUITPY_RANDOM=1 $
-DCIRCUITPY_ROTARYIO=1 $
-DCIRCUITPY_RTC=1 $
-DCIRCUITPY_SAFE_RESTART_WORD=0xDEADBEEF $
-DCIRCUITPY_SAMD=0 $
-DCIRCUITPY_SOFTWARE_SAFE_MODE=0x0ADABEEF $
-DCIRCUITPY_STAGE=0 $
-DCIRCUITPY_STORAGE=1 $
-DCIRCUITPY_STRUCT=1 $
-DCIRCUITPY_SUPERVISOR=1 $
-DCIRCUITPY_TIME=1 $
-DCIRCUITPY_TOUCHIO=1 $
-DCIRCUITPY_TOUCHIO_USE_NATIVE=1 $
-DCIRCUITPY_UHEAP=0 $
-DCIRCUITPY_USB_HID=1 $
-DCIRCUITPY_USB_MIDI=1 $
-DCIRCUITPY_USTACK=0 $
-DEXPRESS_BOARD $
-DEXTERNAL_FLASH_DEVICES="S25FL216K, GD25Q16C" $
-DEXTERNAL_FLASH_DEVICE_COUNT=2 $
-DFFCONF_H=$$"lib/oofatfs/ffconf.h$$" $
-DLONGINT_IMPL_MPZ -Os $
-DNDEBUG $
-DSAMD21 $
-DSPI_FLASH_FILESYSTEM=1 $
-DUSB_AVAILABLE $
-D__SAMD21G18A__

warnings = $
-Wall $
-Wcast-align $
-Wdouble-promotion $
-Werror $
-Werror-implicit-function-declaration $
-Wfloat-equal $
-Wmissing-format-attribute $
-Wnested-externs $
-Wno-deprecated-declarations $
-Wno-endif-labels $
-Wno-error=lto-type-mismatch $
-Wshadow $
-Wsign-compare $
-Wstrict-prototypes $
-Wundef $
-Wunreachable-code $
-Wwrite-strings

cflags_pre = $
--param max-inline-insns-single=500 $
-fdata-sections $
-ffunction-sections $
-finline-limit=60 $
-flto -flto-partition=none $
-fno-strict-aliasing $
-fshort-enums $
-fsingle-precision-constant $
-mabi=aapcs-linux $
-mcpu=cortex-m0plus $
-mfloat-abi=soft $
-msoft-float $
-mthumb $
-nostdlib $
-std=gnu11

rule compile
  depfile = $out.d
  command = arm-none-eabi-gcc -MM -MF $out.d $includes $defines -c $in -o $out
  description = CC $out

rule gen-qstr-defs-pre
  command = cat ../py/qstrdefs.h $in | $
sed 's/^Q(.*)/"&"/' | $
arm-none-eabi-gcc -E $includes_pre $defines_pre $warnings $cflags_pre - | $
sed 's/^"\(Q(.*)\)"/\1/' > $out
  description = GEN $out

rule gen-qstr-data
  command = python3 ../py/makeqstrdata.py $in > $out
  description = GEN $out

rule gen-version-hdr
  command = python3 ../py/makeversionhdr.py $out
  description = GEN $out

build objs/main.o : compile ../main.c
build genhdr/qstrdefs.preprocessed.h : gen-qstr-defs-pre $
../ports/atmel-samd/qstrdefsport.h $
../ports/atmel-samd/build-feather_m0_express/genhdr/qstrdefs.collected.h
build genhdr/qstrdefs.enum.h : gen-qstr-data genhdr/qstrdefs.preprocessed.h
build version/genhdr/mpversion.h: gen-version-hdr

tannewt commented 4 years ago

Yesterday I read through the short ninja build manual and wonder if it's worth using the depfile feature that automatically discovers required header files by using gcc -MM -MF $out.d ... to write out all required headers for a given C file. Using depfile could help in several ways:

Besides eliminating the need to specify the implicit header dependencies for an given C file using | foo.h, it also makes the process more robust by ensuring no header files fall through. For example, the current generated ninja.build has:

I definitely want to use depfile for header dependencies. I don't think it'll work for mpversionhdr though because the file doesn't exist until built. I believe you need to be explicit for headers that are generated.

I've been experimenting by directly modifying the build.ninja file and can get past the genhdr/qstrdefs.enum.h error, but I don't trust the generated file is corrected because I haven't yet corrected for all the modified paths in the $include_pre, etc. Which is worrying: the ad-hoc sed commands, clearly are not fault tolerant and I think the sed functionality should be merged into the python later. What do you think?

The sed stuff should definitely be merged into one of the QSTR build scripts.

tannewt commented 4 years ago

@omsai I pushed a new version that completes the QSTR stuff. Unfortunately, the next step is evolving the dynamic linkerscript generation that SAMD does.

durapensa commented 3 years ago

Zephyr's multiple repo & build management tool, west, may be useful. It manages both ninja & cmake builds, and has a nice clean way to manage multiple git repos. https://github.com/zephyrproject-rtos/west Please disregard this comment if if it's not useful.

tannewt commented 3 years ago

Thanks for the tip @durapensa! Can it do ninja without cmake? I don't really know want cmake to be a piece we use.

durapensa commented 3 years ago

@tannewt Peeking at the west docs, it uses cmake to generate ninja build files & board files. But since it's Python, that part can probably be replumbed. There's also a flashing component; could that be handy for priming/repairing boards for usb disk?

omsai commented 3 years ago

@tannewt Can you again suggest some representative hardware target devices to focus this build system retooling effort? I don't mind buying several devices with bare minimum features if helps with quicker wins. A power outage damaged my Metro M4 Airlift which was plugged into a wall outlet at the time. Maybe a device that you're also using and that's still in production?

tannewt commented 3 years ago

@tannewt Peeking at the west docs, it uses cmake to generate ninja build files & board files. But since it's Python, that part can probably be replumbed. There's also a flashing component; could that be handy for priming/repairing boards for usb disk?

Flashing is interesting but not very urgent. Most CircuitPython boards have UF2 bootloaders that are easy to manage separately. The main reason I'd like to rework our build system is to simplify configuration of different boards and generate a single build graph with all of the different configurations together. That will allow us to iterate towards sharing build artifacts across boards.

@tannewt Can you again suggest some representative hardware target devices to focus this build system retooling effort? I don't mind buying several devices with bare minimum features if helps with quicker wins.

I want to caution against you taking this on. I have a lot of ideas about how I want to do it and don't want you to waste your time and money implementing something different than what I'm thinking. We could video chat about it all if you still want to try.

Sorry to hear about your Metro! The ESP32-S2 metro is what I'd recommend as a replacement.

omsai commented 3 years ago

Summary of the project discussion at Friday's Deep Dive video https://youtu.be/Vlxf8FRtV30?t=6418 for others following along this issue:

Long term goal of python writes a single build.ninja file and ninja builds everything. Usually when we build a particular board, there is a heirarchical graph of Makefiles that is used to build that specific case. What we instead want is a single build.ninja file that builds all of the boards at once and share information between boards in the future. For example we compile a file many times over for a Cortex M0, but there are several Cortex M0s that and it would for which it would be nice to share that file.
Recent goal to unify the way we store information about flash chips into "cascade TOML".
1. The RP2040 https://github.com/adafruit/circuitpython/tree/main/ports/raspberrypi boot ROM loads 256 bytes https://github.com/adafruit/circuitpython/blob/main/ports/raspberrypi/bs2_default_padded_checksummed.S from flash by clocking slowly and reading slowly into RAM and execute if it checksums successfully. Currently these bytes are the default single-bit width instead of QSPI
2. The IMX https://github.com/adafruit/circuitpython/tree/main/ports/mimxrt10xx does a similar thing but instead of loading instructions off the flash they load C structs that have configuration settings, for example https://github.com/adafruit/circuitpython/blob/main/ports/mimxrt10xx/boards/imxrt1010_evk/flash_config.c
3. The original way we managed flash is in https://github.com/adafruit/circuitpython/blob/main/supervisor/shared/external_flash/devices.h
4. Would be great to have a https://toml.io system specifying:
  1. The microcontroller chip on the board
  2. Then sub-tree with specializions for the microcontroller all the way down to Coretex M0.
  3. Finally, knowing that Coretex M0, you would have certain compiler settings, etc
  4. Such configuration compilers exist, but are not in an end-result agnostic format.
  5. Could have "cascade TOML". For example:
```
flash.template.toml  # Fake version of the final flash / MRAM object.
cascade.yoml          # Specify the fields.  e.g. shared bindings/busio:
                               # busio could be a flag.

{memory_type}/       # Memory type can be flash or MRAM.
   {manufacturer}/
       {flash_family}/
           {flash_sku}.toml

# Within every level of this we would have a TOML file as well.
# which would set some fields of the flash.template.toml thing
# and so such files have settings that apply to everything
# downstream of it.  For example all flash families of gigadevice
# would have these fields:
#
{memory_type}/gigadevice/gigadevice.toml
#
# It would be an error if something more specific defines
# something at the top level.  There would be no overriding!
# Only a way to share, but not override.
```
  6. We want to minimize the logic in the build system and move it to TOML for configuring with a small python core that does the translation to ninja.
  7. Then could run a command like cascadetoml --flash_sku=GD2156JV to find the flash_sku.toml starting from the highest level TOML and gathering all the settings. This could be piped: cascadetoml --flash_sku=GD2156JV | python devices.h.py to translate the TOML structure into the final output. We would have 3 different outputs for CircuitPython:
    1. devices.h
    2. (RP2040 little chunk of assembly)
    3. (IMX version of that)
  8. There is also flash related board config. Example settings: QSPI? Max frequency for board? flash_sku (could be a list for boards with multiple TOML). Don't know if we need pinout as well; would need to take a look.
  9. Idea behind using flash.template.tomlis instead of using TOML validators we want to be strict about the shape of the TOML. You need to be able to verify that keys pull requested in are valid, so having the fake version to see hopefully catches a lot of the errors. Put comments in the field explaining what it is and the exactly the way it works.

To build: board -> mcu sku -> soc -> cpu core

Could put the flash things either under https://github.com/adafruit because it will be on adafruit time or it will go into https://github.com/baremetal-ninja

There's a nice python library called tomlkit that explicitly preserves comments which would be handy for programmatic refactoring.

omsai commented 3 years ago

@tannewt I can't make today's weekly meeting, but am interested in the next steps of ideas for milestones, a minimum viable product, and to pencil in dates for myself to meet those goals; I'm a fan of setting weekly goals with a few weeks sprinkled in as buffer. I feel like I have enough background now to work independently on some of these things, but in the call you mentioned maybe working on some of this during your week on Adafruit time so I thought it would make sense to coordinate more closely.

Edit: I completely forgot to mention that it would be of particular interest to think about how tasks can be split up among multiple people / multiple GitHub issues because there were some enthusiastic people in the Deep Dive video call interested in contributing.

tannewt commented 3 years ago

Thanks for these notes @omsai! My plan with the flash is to do the flash toml bits, cascade toml and generators for it. I don't think I'll go so far as to do the boards.

I'm not sure what a good small step for this is. One thing I'd do later is optimize across boards. For now we could just generate a giant ninja file to build everything separately. That still could be tricky because we likely want to factor out some of the compiler and board settings as we do that.

This task feels like the initial work needs to be done by one person and once that skeleton is done, others can help it evolve. In particular, once we have cascade toml and the starts of the files, we can parallelize gathering info about more mcus/flashes.

tannewt commented 3 years ago

Ok WIP repos are here: https://github.com/adafruit/cascadetoml and https://github.com/adafruit/nvm.toml

omsai commented 3 years ago

Two questions:

Should I still work on the feather_m0_express in the top comment for building out the ninja script? Or is there reason to merge the years' worth of changes from main back to the ninja branch?

$ git clone -b ninja https://github.com/tannewt/circuitpython.git
$ cd circuitpython
$ git log --oneline origin/main..ninja
8d63932dd (HEAD -> ninja, origin/ninja) Fix qstr generation. Fails with linker script now
d2838e585 Merge remote-tracking branch 'adafruit/master' into ninja
55fb4f3f8 More qstr progress
8f66cf767 Build runs until needing qstr

This may be early to try using, but is the intention that the cascade command be fed the terminal leaf TOML file? My python 3.7.3 on Debian complains when trying it:

$ git clone https://github.com/adafruit/cascadetoml.git
$ git clone https://github.com/adafruit/nvm.toml.git
$ cd cascadetoml
$ python3 cascadetoml.py check ../nvm.toml/
$ echo $?
0
$ python3 cascadetoml.py cascade ../nvm.toml/flash/gigadevice/GD25Q16C.toml cascade
(error below)

Traceback (most recent call last):
  File "cascadetoml.py", line 206, in <module>
    app()
  File "/home/omsai/.local/lib/python3.7/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/omsai/.local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/omsai/.local/lib/python3.7/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "cascadetoml.py", line 74, in cascade
    if not parent.is_relative_to(root.parent):
AttributeError: 'PosixPath' object has no attribute 'is_relative_to'

tannewt commented 3 years ago

Two questions:

Should I still work on the feather_m0_express in the top comment for building out the ninja script? Or is there reason to merge the years' worth of changes from main back to the ninja branch?

I'd probably start with the latest main and merge in the files from the old branch. I don't think there is much history there worth saving. I believe the QSTR stuff may have changed a bit. It's probably one of the larger things to move over.

This may be early to try using, but is the intention that the cascade command be fed the terminal leaf TOML file? My python 3.7.3 on Debian complains when trying it:

That's one way to do it:

$ cascadetoml cascade files flash/gigadevice/GD25Q16C.toml

# Data for path: flash/gigadevice/GD25Q16C.toml
# Data inferred from the path: {technology}/{manufacturer}/{sku}.toml
technology = "flash"
manufacturer = "gigadevice"
sku = "GD25Q16C"

# Data from flash/flash.toml
supports_fast_read = true

# Data from flash/gigadevice/gigadevice.toml
start_up_time_us = 5000
manufacturer_id = 0xc8
memory_type = 0x40
max_clock_speed_mhz = 104 # if we need 120 then we can turn on high performance mode
quad_enable_bit_mask = 0x02
has_sector_protection = false
supports_qspi = true
supports_qspi_writes = true
single_status_byte = false

# Data from flash/gigadevice/GD25Q16C.toml
# Settings for the Gigadevice GD25Q16C 2MiB SPI flash.
# Datasheet: http://www.gigadevice.com/datasheet/gd25q16c/
total_size = 0x200000 # 2 MiB
capacity = 0x15
write_status_register_split = false

The other way is: cascadetoml cascade filter sku=\"GD25Q64C\"

tannewt commented 3 months ago

I don't want to move to ninja anymore. Ideally it'd be an all-Python build system. No reason to have an issue for this work though.

adafruit / circuitpython

Move to ninja build system #2902