arduino-cli compile with --output-dir flag set fails to build *.with_bootloader.hex file.

[a7hybnj2]

So, I have been using MightyCore for years and today I saw there was an update to 2.2.2 but I also saw there was an arduino cli update so I did both at once and everything broke. I realized that my arduino-cli upload command was trying to upload the with_bootloader.hex file which wasn't even being compiled out anymore.

So, I downgraded to my previous versions of both: arduino-cli Version: 0.32.2 && MightyCore:avr 2.1.3

So, it tried again and it worked but when I went through the logs it looks like it is uploading the with_bootloader.hex but at least the old version would compile out both.

I don't actually know where the error is because I think 2.2.2 introduced a new avrdude also.

Anyway my data:

programmer = usbtinyisp
compile: arduino-cli compile --fqbn $(fqbn) --output-dir $(output-dir) --verbose
upload: arduino-cli upload --fqbn $(fqbn) --input-dir $(output-dir) --programmer $(programmer) --verify --verbose

I see there is a flag for arduino-cli upload of --input-file but I have never specified it before. And I don't know why after updating to 2.2.2 and the with_bootloader file wasn't being generated anymore that it wouldn't just pull the *.hex file in the build dir.

[a7hybnj2]

I am beginning to think this is something with the arduino-cli and not mightycore but I will leave it here in the event someone has a similar issue or can confirm or deny:

So I just spelled out the command in full and I still get it trying to use with_bootloader.hex

arduino-cli upload --fqbn MightyCore:avr:324:bootloader=no_bootloader,eeprom=keep,pinout=standard,variant=modelPB,BOD=2v7,LTO=Os,clock=16MHz_external --input-file build/324Blink.ino.hex --programmer usbtinyisp --verify --verbose
"/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/tools/avrdude/7.1-arduino.1/bin/avrdude" "-C/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.1/avrdude.conf" -v  -patmega324pb -cusbtiny  "-Uflash:w:build/324Blink.ino.with_bootloader.hex:i"

avrdude: Version 7.1-arduino.1
         Copyright the AVRDUDE authors;
         see https://github.com/avrdudes/avrdude/blob/main/AUTHORS

         System wide configuration file is /Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.1/avrdude.conf
         User configuration file is /Users/a7hybnj2/.avrduderc
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : usb
         Using Programmer              : usbtiny
avrdude: usbdev_open(): found USBtinyISP, bus:device: 005:003
avrdude ioerror() OS error: file build/324Blink.ino.with_bootloader.hex is not readable: No such file or directory

avrdude done.  Thank you.

Failed programming: uploading error: exit status 1
make: *** [upload] Error 1

Let me know what you think.

[a7hybnj2]

Okay, to make sure it wasn't something that has already been fixed I cleared all cores and arduino stuff from my computer. I now only have:

arduino-cli Version: 0.33.0 Commit: ca60d4b4 Date: 2023-05-29T15:18:18Z

arduino-cli core list
ID             Installed Latest Name      
MightyCore:avr 2.2.2     2.2.2  MightyCore

Somewhere something is automatically generating this "-Uflash:w:build/324Blink.ino.with_bootloader.hex:i" and it doesn't seem like I can override it either.

[a7hybnj2]

Okay, this is my last update until someone else chimes in. So, I removed the build-dir and input-file options to see where the default build directory was. I then went to that directory and deleted all the previously compiled files. I then did a fresh compile, listed below, which appears to contain NO with_bootloader.hex files but when I go to upload it, listed further below, it pulls the withbootloader.hex file to use to upload. I have cleared every cache and conf and yaml associated with arduino and cores and this appears to be the default behaviour ¯\_(ツ)/¯

arduino-cli compile --fqbn MightyCore:avr:324:bootloader=no_bootloader,eeprom=keep,pinout=standard,variant=modelPB,BOD=2v7,LTO=Os,clock=16MHz_external 
--verbose
FQBN: MightyCore:avr:324:bootloader=no_bootloader,variant=modelPB
Using board '324' from platform in folder: /Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2
Using core 'MCUdude_corefiles' from platform in folder: /Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2

Detecting libraries used...
/Users/a7hybnj2/Library/Arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++17 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -w -x c++ -E -CC -mmcu=atmega324pb -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_ATmega324 -DARDUINO_ARCH_AVR -I/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/cores/MCUdude_corefiles -I/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/variants/standard /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/sketch/324Blink.ino.cpp -o /dev/null
Generating function prototypes...
/Users/a7hybnj2/Library/Arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++17 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -w -x c++ -E -CC -mmcu=atmega324pb -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_ATmega324 -DARDUINO_ARCH_AVR -I/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/cores/MCUdude_corefiles -I/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/variants/standard /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/sketch/324Blink.ino.cpp -o /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/preproc/sketch_merged.cpp
/Users/a7hybnj2/Library/Arduino15/packages/builtin/tools/ctags/5.8-arduino11/ctags -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/preproc/sketch_merged.cpp
Compiling sketch...
/Users/a7hybnj2/Library/Arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++17 -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -MMD -mmcu=atmega324pb -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_ATmega324 -DARDUINO_ARCH_AVR -I/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/cores/MCUdude_corefiles -I/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/variants/standard /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/sketch/324Blink.ino.cpp -o /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/sketch/324Blink.ino.cpp.o
Compiling libraries...
Compiling core...
Using precompiled core: /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/cores/2ee9b738610332e7c4207978ea960262/core.a
Linking everything together...
/Users/a7hybnj2/Library/Arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-gcc -w -Os -Wl,--gc-sections,--section-start=.FAR_MEM1=0x10000 -mmcu=atmega324pb -o /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.elf /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/sketch/324Blink.ino.cpp.o /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/../../cores/2ee9b738610332e7c4207978ea960262/core.a -L/private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800 -lm
/Users/a7hybnj2/Library/Arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-objcopy -O ihex -j .eeprom --set-section-flags=.eeprom=alloc,load --no-change-warnings --change-section-lma .eeprom=0 /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.elf /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.eep
/Users/a7hybnj2/Library/Arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-objcopy -O ihex -R .eeprom /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.elf /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.hex
chmod +x /Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/scripts/create_disassembler_listing.sh
/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/scripts/create_disassembler_listing.sh /Users/a7hybnj2/Library/Arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-objdump "--disassemble --source --line-numbers --demangle --section=.text" /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.elf /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino_atmega324pb_16000000L.lst

/Users/a7hybnj2/Library/Arduino15/packages/arduino/tools/avr-gcc/7.3.0-atmel3.6.1-arduino7/bin/avr-size -A /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.elf
Sketch uses 2590 bytes (7%) of program storage space. Maximum is 32768 bytes.
Global variables use 184 bytes (8%) of dynamic memory, leaving 1864 bytes for local variables. Maximum is 2048 bytes.

Used platform  Version Path                                                                    
MightyCore:avr 2.2.2   /Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2

arduino-cli upload --fqbn MightyCore:avr:324:bootloader=no_bootloader,eeprom=keep,pinout=standard,variant=modelPB,BOD=2v7,LTO=Os,clock=16MHz_external -
-programmer usbtinyisp --verify --verbose
"/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/tools/avrdude/7.1-arduino.1/bin/avrdude" "-C/Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/avrdude.conf" -v  -patmega324pb -cusbtiny  "-Uflash:w:/private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.with_bootloader.hex:i"

avrdude: Version 7.1-arduino.1
         Copyright the AVRDUDE authors;
         see https://github.com/avrdudes/avrdude/blob/main/AUTHORS

         System wide configuration file is /Users/a7hybnj2/Library/Arduino15/packages/MightyCore/hardware/avr/2.2.2/avrdude.conf
         User configuration file is /Users/a7hybnj2/.avrduderc
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : usb
         Using Programmer              : usbtiny
avrdude: usbdev_open(): found USBtinyISP, bus:device: 005:003
         AVR Part                      : ATmega324PB
         Chip Erase delay              : 55000 us
         PAGEL                         : PD7
         BS2                           : PA0
         RESET disposition             : possible i/o
         RETRY pulse                   : SCK
         Serial program mode           : yes
         Parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                           Block Poll               Page                       Polled
           Memory Type Alias    Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- -------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom                 65    10   128    0 no       1024    4      0  9000  9000 0xff 0xff
           flash                  33     6   256    0 yes     32768  128    256  4500  4500 0xff 0xff
           lfuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
           hfuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
           efuse                   0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
           lock                    0     0     0    0 no          1    1      0  9000  9000 0x00 0x00
           signature               0     0     0    0 no          3    1      0     0     0 0x00 0x00
           calibration             0     0     0    0 no          1    1      0     0     0 0x00 0x00

         Programmer Type : USBtiny
         Description     : USBtiny simple USB programmer, https://learn.adafruit.com/usbtinyisp
avrdude pgm_default() error: programmer operation not supported

avrdude: using SCK period of 10 usec
avrdude: AVR device initialized and ready to accept instructions
avrdude: device signature = 0x1e9517 (probably m324pb)
avrdude: Note: flash memory has been specified, an erase cycle will be performed.
         To disable this feature, specify the -D option.
avrdude: erasing chip
avrdude: using SCK period of 10 usec
avrdude: reading input file /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.with_bootloader.hex for flash
         with 2590 bytes in 1 section within [0, 0xa1d]
         using 21 pages and 98 pad bytes
avrdude: writing 2590 bytes flash ...

Writing | ################################################## | 100% 4.46s

avrdude: 2590 bytes of flash written
avrdude: verifying flash memory against /private/var/folders/n1/qjm83k_105312fpt4p66q1rc0000gn/T/arduino/sketches/2A83BFD6B6D2FE751DCF1BC0B79BB800/324Blink.ino.with_bootloader.hex

Reading | ################################################## | 100% 2.40s

avrdude: 2590 bytes of flash verified

avrdude done.  Thank you.

[a7hybnj2]

Okay, so a new thing started happening when I would compile with --output-dir specified. I would get a file called {sketch_path} in my main directory. I see sketch_path which I guess is supposed to expand into a filename other than literal {sketch_path}

The contents of the file {sketch_path} appear to be the same as 324Blink.ino_atmega324pb_16000000L.lst and you can see in the build directory there is not file with_bootloader.hex being copied over. The with_bootloader.bin is being copied but the .hex isn't being generated.

I also see in the AVR Uploader/Programmers tools that it is appending the with_bootloader.hex onto one of the tools.

[a7hybnj2]

Okay, I think this boils down to a much simpler problem. The with_bootloader.hex file is being used because I am specifying a programmer but it doesn't actually contain a bootloader, I am fine with that.

It seems like the problem is in my image above, when you specify --output-dir it doesn't compile out the with_bootloader.hex. It doesn't even show up in the hashed build directory. But, when you don't specify --output-dir it builds to the hashed build directory and the with_bootloader.hex is created/compiled.

I am going to rename this issue to better reflect the actual issue.

[a7hybnj2]

I just rolled back to arduino-cli version 0.25.1 and all the correct files were built and moved to the --output-dir but I don't necessarily think this is an arduino-cli issue becasue using the arduino:avr:uno core with 0.33.0 does indeed copy all the files including the *.with_bootloader.hex to the specified --output-dir.

I have officially exhausted my ability to diagnose the issue further. Thanks irc:timemage for following me get this far.

[a7hybnj2]

Me again...

Testing arduino:avr:uno gave me the idea of turning on the bootloader arduino-cli compile --fqbn MightyCore:avr:324:bootloader=uart0,eeprom=keep,pinout=standard,variant=modelPB,BOD=2v7,LTO=Os,clock=16MHz_external --output-dir $(output-dir) --verbose which does copy and use the .with_bootloader.hex file. Just to reiterate my understanding, the with_bootloader is a merged file between .ino.hex and the bootloader, the bootloader of no_bootloader is empty.hex. But, you still need to have the file called with_bootloader.hex because it appears to be a hard coded appended suffix.

So it looks like it has to do with the merged_output being generated/exported/deleted

adding --build-property "build.export_merged_output=true" to the compile command fixes things. I have never needed to do this before and the {sketch_path} file is still being created but that might help further narrow down the issue.

arduino-cli compile --verbose --fqbn MightyCore:avr:324:bootloader=no_bootloader,eeprom=keep,pinout=standard,variant=modelPB,BOD=2v7,LTO=Os,clock=16MHz_external --output-dir /tmp/blank-output/ --build-property "build.export_merged_output=true"

This seems like an appropriate work around for me while this change is getting looked into, hopefully by someone that knows what all the crazy linking and build recipes do.

[per1234]

I would get a file called {sketch_path} in my main directory. I see sketch_path which I guess is supposed to expand into a filename other than literal {sketch_path}

This bug is now tracked by the Arduino CLI developers at https://github.com/arduino/arduino-cli/issues/2340. There is no need to comment there. I'm only adding this comment to notify affected users that there is a formal bug report about it in the appropriate issue tracker.

[MCUdude]

I'm closing this issue because it's related to the arduino-cli and not MightyCore.