New directory structure and file name convention

Carried over from https://github.com/stefanrueger/urboot/issues/19 @MCUdude @mcuee @SpenceKonde feel free to comment

[edited to remove _fr frills from options by making them automatically included if there is space]

Some thoughts about what to offer below the mcus/<mcu> level. Note I plan a slightly different structure/naming for cores (will comment later, but basically to make it easier to cope with boards.txt file).

Would like to have WDTO as first directory level (not sure I want to populate that, but at least conceptionally it should be there); max three directory names would be
- watchdog_timeout_+512 ms
- watchdog_timeout_1024 ms
- watchdog_timeout_2048 ms
  - Next directory level with up to 21 different oscillator modes (again, probably only populate autobaud, external and internal-4%, ... internal+4%). Is it OK to use % in directory names?
- autobaud
- external_oscillator_x
- internal_oscillator_a-9% ...
- internal_oscillator_h-1%
- internal_oscillator_i
- internal_oscillator_j+1% ...
- internal_oscillator_q+9%
  - Next directory level would be F_CPU (unless it's autobaud); here only nominal values for internal oscillators (8 MHz, 1 MHz and 128 kHz for ordinary parts or 9.6 MHz, 4.8 MHz, ... for ATtiny13); the deviation has been selected in the level above
- fcpu_+0m1250
- fcpu_+0m2304
- fcpu_+0m2500
- fcpu_+0m4608
- fcpu_+0m5000
- fcpu_+0m9216
- fcpu_+1m0000
- fcpu_+1m3824
- fcpu_+1m5000
- fcpu_+1m8432
- fcpu_+2m0000
- fcpu_+2m3040
- fcpu_+2m5000
- fcpu_+2m7648
- fcpu_+3m0000
- fcpu_+3m6864
- fcpu_+4m0000
- fcpu_+4m6080
- fcpu_+5m0000
- fcpu_+5m5296
- fcpu_+6m0000
- fcpu_+7m3728
- fcpu_+8m0000
- fcpu_+9m2160
- fcpu_10m0000
- fcpu_11m0592
- fcpu_12m0000
- fcpu_14m7456
- fcpu_16m0000
- fcpu_18m4320
- fcpu_20m0000
- fcpu_22m1184
- fcpu_24m0000
  - Next level baud rates (unless it's autobaud), eg,
- br_++19k2
- br_++28k8
- br_++38k4
- br_++57k6
- br_++76k8
- br_+115k2
- br_+125k0
- br_+230k4
- br_+250k0
- br_+460k8
- br_+500k0
- br_+576k0
- br_1000k0
  - Next level would be the I/O port; it would be either a list of UARTs (for parts that have UARTs) or a list of SWIO combos (for parts that don't) but always with rx/tx line, eg, for the ATtiny841 below (the thing is that some fcpu/br combos can only be implemented with SWIO, but the "I/O port" name would still be UART...)
- uart0_alt1_rxb2_txa7
- uart0_rxa2_txa1
- uart1_rxa4_txa5
  - Next level would be the board configuration (LED Pin, LED polarity, CS line of external SPI flash for dual boot)
- no-led (actually not board/core specific, just a bootloader for the part, but every board/core will have that)
- lednop
- template
- led-b5
- led+b7
- led+b5_csd5_dual
  - Then, finally phew, the bootloaders; their name would be unique, even without the directory structures and they would look sth like urboot_m328p_w1_8m0x_115k2_swio_rxd0_txd1_no-led_ee_ce_hw_stk500.hex, urboot_t167_autobaud__uart0_rxa0_txa1_p.hex or urboot_t13a_4m8h_38k4_swio_rxb1_txb0_led+b2_p_ee_ce.hex; notice the padding + form the directories are gone, the AVRDUDE short name of the part us used and the letter after the frequency that indicates oscillator type and whether external or internal and if so how much it's above or below the nominal value (h is internal oscillator - 1%); notice also that the method to create the baud rate for UART0 for the ATmega329P could be swio
- p reset vector protection for vector bootloader
- ee EEPROM r/w
- ce can emulate chip erase (and does so from top to bottom)
- dual dual boot
- hw hardware supported bootloader
- stk500 old STK500v1 protocol, compatible with avrdude -c arduino, not recommended will be deprecated
  - There are a number of frills, non-essential properties of a bootloader, eg, quicker exit on errors, that will secretly be compiled in if there is space within the current page limit. Also only planning on certain combinations
- (nothing) for the space conscious
- p
- p_ce
- p_ee
- p_ee_ce
- p_ee_ce_dual
- hw (only in parts with 256 byte bootsection)
- ce_hw (only in parts with 256 byte bootsection, may not be available as too big for 256 bytes)
- ee_ce_hw
- ee_ce_hw_stk500 ("drop-in replacement" for optiboot)
- ee_ce_dual_hw

Just a few general comments. 1) The directory structure is quite complicated, I guess you want to accomodate many use cases so it is probably difficult to avoid the number of layers.

Most likley @MCUdude and @SpenceKonde will only select a subset for their respective Arduino Core projectes.

2) Naming of baudrate, somehow I feel br_++19k2 is not as good as br_19200.

3) watchdog_timeout_xxxxx ms --> Good to have 8192ms option when it is possible. 8192ms is sometimes useful if there is no auto-reset mechanism available for the setup. This becomes less important for me once https://github.com/avrdudes/avrdude/pull/1309 is merged. But it may still be useful sometimes.

https://github.com/avrdudes/avrdude/pull/1309

Thanks @mcuee: great shouts. The 2048 ms are actually intended for reset by hand. Works quite comfortably for me. This is how I do it: First press the reset button of the board and release it; then start avrdude by pressing enter of the previously typed command line within the WDTO time. 8 s is quite a long time to wait for the application to come on in normal circs when resetting manually (in normal operation, without intention to flash)

Above structure under mcus/ is meant for manual selection. The idea is that a user rocks up with a part, maybe they want to use the internal 8 MHz and have measured it to be 2.2% too slow, they could then go down the tree via internal_oscillator_g-2% and finally get to the bootloader they need. They will in the leaf directory have no more than a handful of choices. The '+' padding helps to find the right frequency or baud rate (I find it mildly irritating when numeric values are sorted lexicographically).

Now to the proposed cores directory scheme:

WDTO as first directory level (as above, this might be populated for 1 s and 2 s WDTO)
- watchdog_timeout_+512_ms
- watchdog_timeout_1024_ms
- watchdog_timeout_2048_ms
  - Next directory level with max 3 different oscillator modes
- autobaud
- external_oscillator
- internal_oscillator
  - Next directory level would be F_CPU (unless it's autobaud); here the actual values for percentage variations of internal oscillators would be used (1000000, 1010000, ..., 960000, 970000, ...)
- fcpu_1000000
- fcpu_16000000 ...
- fcpu_8000000 ...
  - Next level baud rates (unless it's autobaud), eg,
- br_1000000
- br_115200
- br_125000
- br_19200
- br_230400
- br_250000
- br_28800
- br_38400
- br_460800
- br_500000
- br_57600
- br_576000
- br_76800
  - Next level would be the I/O port; as above
- uart0_alt1_rxb2_txa7
- uart0_rxa2_txa1
- uart1_rxa4_txa5
  - Next level would be the board configuration as above
- no-led
- lednop
- template
- led-b5
- led+b7
- led+b5_csd5_dual
  - The bootloaders would not have unique names, just short names like below
- urboot.hex
- urboot_p.hex
- urboot_p_ce.hex
- urboot_p_ee.hex
- urboot_p_ee_ce.hex
- urboot_p_ee_ce_dual.hex
- urboot_hw.hex (only in parts with 256 byte bootsection)
- urboot_ce_hw.hex (only in parts with 256 byte bootsection, may not be available as too big for 256 bytes)
- urboot_ee_ce_hw.hex
- urboot_ee_ce_hw_stk500.hex ("drop-in replacement" for optiboot)
- urboot_ee_ce_dual_hw.hex

This way you can use your beloved simple replacement in boards.txt. I would implement this either as copy, hard or soft link from the corresponding bootloader with a long name under mcus.

I haven't used Arduino for well over a decade; IIRC the boards.txt file contains the max upload flash size that the Arduino environment would use to check against too large sketches. With AVRDUDE v7.1 and avrdude -c urclock this is no longer necessary, as it knows how big the bootloader is. But if the size is still a thing, then hexls -use name.hex gives you the occupancy of the bootloader.

BTW, I recommend use of urboot_p_ee_ce.hex bootloaders; I predict (but don't know) they will almost universally fit into the same number of bytes within a core: 384 for MiniCore (3 pages) and 512 for MightyCore (2 pages), eg.

Once I hear from you @MCUdude, @mcuee, @SpenceKonde it might still be a week or so for that to become reality.

The 2048 ms are actually intended for reset by hand. Works quite comfortably for me. This is how I do it: First press the reset button of the board and release it; then start avrdude by pressing enter of the previously typed command line within the WDTO time. 8 s is quite a long time to wait for the application to come on in normal circs when resetting manually (in normal operation, without intention to flash)

That works if you;re using AVRdude from the command line. But someone using it through the IDE is not going to be doing that, they're going to use the upload button. That takes considerably longer since it has to compile as well. , and 2 seconds is kinda fiddly for that... 8s is not a long time when you have to time a reset to coincide with the start of the compile process, even though there's no delay between the end of compilation and the start of the upload attempt, so you need to watch the upload process a few times, to know what the right moment to release the button is. Then again, any bootloader-using board that isn't using autoreset is going to provide a rather poor user experience

Then again, any bootloader-using board that isn't using autoreset is going to provide a rather poor user experience

I agree. So this 8192ms bootloaders may be more useful to testers like myself and may not be so useful for the end users. Some board may not even have the reset buttons -- so I have to unplug and plug the board to simulate power up reset. 8192ms will be the minimum for this case -- sometimes Windows may be in the way as well by popping up something.

@mcuee @MCUdude @SpenceKonde

Implemented new directory structure and file name convention

The main aims of this are to

Provide a human browseable tree for mcus and boards, eg, list F_CPUs and baud rates in git numerically and have long unique file names for the bootloaders
Create a machine accessible tree for cores directories, eg, the raw F_CPU and baud rate values are used in the directory structure; the bootloader file names are short and not unique
Distinguish between external and internal oscillators
Use the correct list of internal frequencies that can exist after external reset

There is a subtle distinction between bootloaders for MCUs driven by an external oscillator, which tend to have a small (< 0.2%) deviation from the nominal F_CPU, and MCUs running on an internal oscillator, which more often than not exhibit a deviation of 2% or more from the nominal clock frequency. Urboot bootloaders can and typically will differ for internal and external oscillators of the same frequency

For external oscillators, the pre-compiled urboot bootloaders tend to be shorter and deploy higher baud rates using hardware UARTs (but not always) at the cost of potentially higher quantisation errors for the baud rate, up to 2.1%.
For internal oscillators the bootloaders tend to use lower baud rates using software I/O (but not always) and have a baud rate quantisation error of less than 0.5%; there are a number of bootloaders on offer in steps of 1% around the nominal F_CPU so that a particular board can be furnished with a suitable bootloader for the actual internal clock frequency. All in all the aim is to ensure baud rate errors are below 2.5% to facilitate reliable serial communication.

The boards and mcus directories have the following directory structure below the part name:

WDTO as first directory level (currently only for 1 s and 2 s WDTO)
- watchdog_timeout_0.5_s
- watchdog_timeout_1_s
- watchdog_timeout_2_s
  - Next directory level with up to 21 different oscillator modes
- autobaud
- external_oscillator
- internal_oscillator
- internal_oscillator-1%
- internal_oscillator+1% ...
- internal_oscillator-9%
- internal_oscillator+9%
  - Next directory level F_CPU (unless it's autobaud); here only nominal values for internal oscillators (8 MHz, 1 MHz and 128 kHz for ordinary parts or 9.6 MHz, 4.8 MHz, ... for ATtiny13); the deviation has been selected in the level above
- +0m125000_hz
- +0m230400_hz
- +0m250000_hz
- +0m460800_hz
- +0m500000_hz ....
- 14m745600_hz
- 16m000000_hz
- 18m432000_hz
- 20m000000_hz
- 22m118400_hz
- 24m000000_hz
  - Next level baud rates (unless it's autobaud), eg,
- ++19k2_baud
- ++28k8_baud
- ++38k4_baud
- ++57k6_baud
- ++76k8_baud
- +115k2_baud
- +125k0_baud
- +230k4_baud
- +250k0_baud
- +460k8_baud
- +500k0_baud
- +576k0_baud
- 1000k0_baud
  - Next level is the I/O port; it is either a list of UARTs (for parts that have UARTs) or a list of SWIO combos (for parts that don't) but always with rx/tx line, eg, for the ATtiny841 below (the thing is that some fcpu/br combos can only be implemented with SWIO, but the "I/O port" name is still UART...)
- uart0_alt1_rxb2_txa7
- uart0_rxa2_txa1
- uart1_rxa4_txa5
  - Next level is the bootloader configuration in terms of LED Pin, LED polarity, CS line of external SPI flash for dual boot
- no-led (actually not board/core specific, just a bootloader for the part, but every board/core will have that)
- lednop
- led-b5
- led+b7
- led+b5_csd5_dual
- template_dual
  - Finally, the bootloaders; their name is unique, even without the directory structures and they look sth like urboot_m328p_1s_x8m0_115k2_swio_rxd0_txd1_no-led_pr_ee_ce.hex, urboot_t167_1s_autobaud_uart0_rxa0_txa1_lednop_pr.hex or urboot_t13a_1s_h4m8_38k4_swio_rxb1_txb0_led+b2_pr_ee_ce.hex; notice the padding + form the directories are gone, the AVRDUDE short name of the part us used and the letter before the frequency that indicates oscillator type and whether external or internal and if so how much it's above or below the nominal value (a ... h are internal oscillators with deviations of -8% ... -1%, respectively); notice also that the method to create the baud rate for UART0 for the ATmega328P could be swio because the UART exhibits a too high quantisation error for this case.
- pr reset vector protection for vector bootloader
- ee EEPROM r/w
- ce can emulate chip erase (and does so from top to bottom)
- dual dual boot
- hw hardware supported bootloader
- stk500 old STK500v1 protocol, compatible with avrdude -c arduino, not recommended will be deprecated

There are a number of frills, non-essential properties of a bootloader, eg, quicker exit on errors, that will secretly be compiled in if there is space within the current boot section usage. Only the following combinations of above are implemented:

(no appendix, a vector bootloader that may or may not protect the reset vector: it does if there was space left)
pr
pr_ce
pr_ee
pr_ee_ce
pr_ee_ce_dual
hw (only in parts with 256 byte boot section)
ce_hw (only in parts with 256 byte boot section, but unlikely to fit in 256 bytes)
ee_ce_hw
ee_ce_hw_stk500 ("drop-in replacement" for optiboot, but deprecated as too big)
ee_ce_dual_hw

The cores directory scheme:

WDTO as first directory level (currently only for 1 s and 2 s WDTO)
- watchdog_timeout_0.5_s
- watchdog_timeout_1_s
- watchdog_timeout_2_s
  - Next directory level with max 3 different oscillator modes
- autobaud
- external_oscillator
- internal_oscillator
  - Next directory level is F_CPU (unless it's autobaud); the actual values for variations of internal oscillators are used (1000000, 1010000, ..., 960000, 970000, ...)
- 1000000_hz
- 16000000_hz ...
- 8000000_hz ...
  - Next level baud rates (unless it's autobaud), eg,
- 1000000_baud
- 115200_baud
- 125000_baud
- 19200_baud
- 230400_baud
- 250000_baud
- 28800_baud
- 38400_baud
- 460800_baud
- 500000_baud
- 57600_baud
- 576000_baud
- 76800_baud
  - Next level is the I/O port; as above
- uart0_alt1_rxb2_txa7
- uart0_rxa2_txa1
- uart1_rxa4_txa5
  - Next level is the board configuration as above
- no-led
- lednop
- template
- led-b5
- led+b7
- led+b5_csd5_dual
  - The bootloaders do not have unique names, just short names like below
- urboot_atmega328p.hex
- urboot_atmega328p_pr.hex
- urboot_atmega328p_pr_ce.hex
- urboot_atmega328p_pr_ee.hex
- urboot_atmega328p_pr_ee_ce.hex
- urboot_atmega328p_pr_ee_ce_dual.hex
- urboot_atmega328p_hw.hex (only in parts with 256 byte boot section)
- urboot_atmega328p_ce_hw.hex (only in parts with 256 byte boot section, likely to be too big for 256 bytes)
- urboot_atmega328p_ee_ce_hw.hex
- urboot_atmega328p_ee_ce_hw_stk500.hex ("drop-in replacement" for optiboot)
- urboot_atmega328p_ee_ce_dual_hw.hex

Good to have 8192ms option when it is possible

@mcuee @SpenceKonde I haven't implemented that yet. Can do when I have found an easy way to augur the watchdog timeouts from the ATDF files. Not all parts have a 128 kHz WDT frequency and not all have the same pre-scalers.

Good to have 8192ms option when it is possible

@mcuee @SpenceKonde I haven't implemented that yet. Can do when I have found an easy way to augur the watchdog timeouts from the ATDF files. Not all parts have a 128 kHz WDT frequency and not all have the same pre-scalers.

Thanks, With the merging of https://github.com/avrdudes/avrdude/pull/1309, I am less concerned about the lacking of 8192ms WDT timeout urboot bootloader for testing purpose. I just need to use the USB-TTL adapters with either DTR pin or RTS pin (or both). And I have two FT232RL adapters which have the DTR and CTS pins.

FT232RL_AVRISP

Previously, for testing of Optiboot for mega0 AVR (MegaCoreX), Tiny0/1/2 AVR (megaTinyCore) and AVR Dx (DxCore), Optiboot FW with 8192ms WDT timeout was important for me since some of the boards I have do not have automatically reset circuitry built-in. But now at least I can use still use the external FT232RL for testing in that case.

Good to have 8192ms option when it is possible

@mcuee @SpenceKonde I haven't implemented that yet. Can do when I have found an easy way to augur the watchdog timeouts from the ATDF files. Not all parts have a 128 kHz WDT frequency and not all have the same pre-scalers.

As @SpenceKonde mentioned before, Then again, any bootloader-using board that isn't using autoreset is going to provide a rather poor user experience, I think this is a lower priority.

@SpenceKonde For ATtinyCore (https://github.com/SpenceKonde/ATTinyCore), does the tiny Class AVRs of concerns support 8192ms WDT time out?

Another thing, if I look into the README.md of ATTinyCore, there are a few Tiny AVRs without bootloader. 1) ATtiny26 -- still in production, no urboot bootloader as of now 2) ATTiny2313 and ATtiny2313A/4313 -- urboot bootloader available 3) ATtiny43U -- urboot bootloader available

ATtiny26 -- still in production, no urboot bootloader as of now

And there is a good reason for this; the urboot makefile tells you when trying:

$ make MCU=attiny26
urboot-gcc: attiny26 not supported (AVR2 core without spm instruction)

Following @MCUdude's comment I've increased the spread and range of internal-frequency variations. Also renamed the directory files for oscillator selection for mcus and boards to sth like

      autobaud
      external_oscillator_x
      internal_oscillator_c-7.50%
      internal_oscillator_d-6.25%
      internal_oscillator_e-5.00%
      internal_oscillator_f-3.75%
      internal_oscillator_g-2.50%
      internal_oscillator_h-1.25%
      internal_oscillator_i
      internal_oscillator_j+1.25%
      internal_oscillator_k+2.50%
      internal_oscillator_l+3.75%
      internal_oscillator_m+5.00%
      internal_oscillator_n+6.25%
      internal_oscillator_o+7.50%

This is that the frequencies are listed in numerical order. The cores structure is still the same (just the actual fcpu directory entries changed).

I might increase the range to +/- 8.75%, which means that if the actual frequency is off by, say, 10.5% that part would still be able to work with the closest one of 8.75%.

I am running into my git-fu limits though (or git is running into limits with the millions of files). The last git add, git commit, git push has taken the best part of 4 hours:

$ time git add mcus boards cores

real    175m10.916s
user    166m32.791s
sys 2m14.044s

$ time git commit -m"Increase spread and range of internal-frequency variations"    

... [hundreds of thousands of lines including, eg,]
 rename mcus/attiny88/watchdog_2_s/{internal_oscillator-1% => internal_oscillator_h-1.25%}/+1m000000_hz/+++9k6_baud/swio_rxd7_txd6/led+d0/urboot_t88_2s_h1m0_9k6_swio_rxd7_txd6_led+d0.hex (100%)
 rename mcus/attiny88/watchdog_2_s/{internal_oscillator-1% => internal_oscillator_h-1.25%}/+1m000000_hz/+++9k6_baud/swio_rxd7_txd6/led+d0/urboot_t88_2s_h1m0_9k6_swio_rxd7_txd6_led+d0_pr.hex (100%)
 rename mcus/attiny88/watchdog_2_s/{internal_oscillator-1% => internal_oscillator_h-1.25%}/+1m000000_hz/+++9k6_baud/swio_rxd7_txd6/led+d0/urboot_t88_2s_h1m0_9k6_swio_rxd7_txd6_led+d0_pr_ce.hex (100%)
 rename mcus/attiny88/watchdog_2_s/{internal_oscillator-1% => internal_oscillator_h-1.25%}/+1m000000_hz/+++9k6_baud/swio_rxd7_txd6/led+d0/urboot_t88_2s_h1m0_9k6_swio_rxd7_txd6_led+d0_pr_ee.hex (100%)
 rename mcus/attiny88/watchdog_2_s/{internal_oscillator-1% => internal_oscillator_h-1.25%}/+1m000000_hz/+++9k6_baud/swio_rxd7_txd6/led+d0/urboot_t88_2s_h1m0_9k6_swio_rxd7_txd6_led+d0_pr_ee_ce.hex (100%)
...

real    7m53.279s
user    3m54.341s
sys 3m36.385s

$ time git add README.md

real    0m9.257s
user    0m8.285s
sys 0m1.619s

$ time git commit -m"Update README.md"
[main 0ce50f11d566] Update README.md
 1 file changed, 1 insertion(+), 1 deletion(-)

real    0m9.266s
user    0m8.980s
sys 0m14.978s

$ time git push
Enumerating objects: 2973240, done.
Counting objects: 100% (2973240/2973240), done.
Delta compression using up to 16 threads
Compressing objects: 100% (930017/930017), done.
Writing objects: 100% (2972918/2972918), 583.30 MiB | 5.63 MiB/s, done.
Total 2972918 (delta 1850157), reused 2972639 (delta 1849881)
remote: Resolving deltas: 100% (1850157/1850157), completed with 3 local objects.
To github.com:stefanrueger/urboot.hex.git
   778fdc95e1c8..0ce50f11d566  main -> main

real    3m16.069s
user    0m24.262s
sys 0m7.865s

$ time git status
On branch main
Your branch is up-to-date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    fix-percent.sed

It took 36.71 seconds to enumerate untracked files. 'status -uno'
may speed it up, but you have to be careful not to forget to add
new files yourself (see 'git help status').
nothing added to commit but untracked files present (use "git add" to track)

real    0m39.475s
user    0m26.701s
sys 0m27.643s

I am running into my git-fu limits though (or git is running into limits with the millions of files). The last git add, git commit, git push has taken the best part of 4 hours:

I cloned the urboot.hex repo yesterday with ~40GB of free space on the disk. After 15 minutes or so the computer turned really hot and really slow, and I got tons of OS warnings saying that the amount of free space was critically low. The whole cloning sequence took about an hour or so.

[cloning] took about an hour or so

:smile: Hopefully catching up with the larger range that I just pushed won't take that long (but it won't reduce your disk space for sure). Git took only 18 minutes this time to add the 255,192 new .hex files. Yes, we are pushing the envelope on git; comes from the explosion of the number of parametrised bootloaders, of course.

Maybe I should go back to my template bootloader idea: I already have a number of markers in the bootloader .elf that signify important parameters: different mov rx, rx opcodes (glorified nops) are placeholders for LED blinking operations that set/clear GPI/O pins and/or CS of the external SPI flash memory. An external label in the .elf file tells any interested program where the bootloader loads the bit pattern for the watchdog timer register. Could do the same for the baud rate divider for UARTs or the loop counter for the bit delay for SWIO; these decide the baud rate in relation to F_CPU. Again the bit twiddling for SWIO operations can be recognised by the port operations, and the operations for the USART registers cn be furnished with external labels.

Soooooo, knowing a dozen or so pre-compiled bootloaders per part, a simple program could generate tens of thousands bootloaders within a few seconds (or just the one you currently need within less than a millisecond). No compiler needed. Just the template bootloaders. But that's for another day.

Just one simple idea to get around the git performance issue -- release them as a single zip file in the git repo for the following folde. https://github.com/stefanrueger/urboot.hex/tree/main/mcus

Then you can also add a README file in this folder to document the directory structure.

Just one simple idea to get around the git performance issue -- release them as a single zip file in the git repo for the following folder.

Maybe a tar file would be even better? No compression, so extracting will be faster.

I don't so much worry about git performance on my side. Some bash file just takes as long as it takes. We are talking O(10m) files, so 3 hours for processing them is O(1 ms) per file. Really, really fast when looked at it from this perspective. I can add mcus.tar cores.tar etc files to the repository though that will increase the load on git. And if the repository creates these files a few times, git will remember them all and resolve many more deltas on a git clone. Unless I do a rebase and force push once in a while to make git forget previous versions of pre-compiled bootloaders.

I defo want to keep the directory structure with all the .hex files, so people can browse, select and download individual bootloaders. (I use that myself rather than compiling a bootloader).

I have decided that rather than putting up additional tar files to give the following advice at the bottom of the readme file:

Cloning this repository can take long and create a large directory (some 70 GB). Pro tip: Download a zip file instead from github or execute on the command line

curl -L https://github.com/stefanrueger/urboot.hex/tarball/master >urboot-hex.tar

This creates a roughly 2 GB gzip compressed tar file from the current snapshot.

I defo want to keep the directory structure with all the .hex files, so people can browse, select and download individual bootloaders. (I use that myself rather than compiling a bootloader).

Haha, that is my use case as well. I do not clone this repo as my two computers (an Acer Windows 11 laptop and Mac Mini M1) do not even have enough disk space.

I have decided that rather than putting up additional tar files to give the following advice at the bottom of the readme file:

Cloning this repository can take long and create a large directory (some 70 GB). Pro tip: Download a zip file instead from github or execute on the command line
curl -L https://github.com/stefanrueger/urboot.hex/tarball/master >urboot-hex.tar
This creates a roughly 2 GB gzip compressed tar file from the current snapshot.

This looks like a very good solution.

I guess i haven't been paying attention but the. 70gb in a repo of avr bootloader files caught my attention..... that seems insane and the 2gb tarball Stull seems too big. Try .bz2 instead of zip - it does a markedly better job sometimes (and nearly the same results at other times

Spence Konde Azzy’S Electronics

New products! Check them out at tindie.com/stores/DrAzzy GitHub: github.com/SpenceKonde ATTinyCore: Arduino support for almost every ATTiny microcontroller Contact: @.***

On Wed, Apr 12, 2023, 07:27 mcuee @.***> wrote:

I have decided that rather than putting up additional tar files to give the following advice at the bottom of the readme file https://github.com/stefanrueger/urboot.hex/blob/main/README.md:

Cloning this repository can take long and create a large directory (some 70 GB). Pro tip: Download a zip file instead from github or execute on the command line

curl -L https://github.com/stefanrueger/urboot.hex/tarball/master >urboot-hex.tar

This creates a roughly 2 GB gzip compressed tar file from the current snapshot.

This looks like a very good solution.

— Reply to this email directly, view it on GitHub https://github.com/stefanrueger/urboot.hex/issues/3#issuecomment-1505106877, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTXEW6D6ABSC5X522C6UYDXA2GRZANCNFSM6AAAAAAV7UQE7Y . You are receiving this because you were mentioned.Message ID: @.***>

Try .bz2 instead of zip

Well, tell github :smile: . The suggested curl command of github's master tarball results in a gzipped tar file on your stdout. It's up to you to recode that into your favourite compression or extract just what you want on the fly, eg, | tar xfz - stefanrueger-urboot.hex-c0b7524/cores/microcore/ ...

But yes, the corresponding tar.bz2 file would only be 700 MB in contrast to the gzipped tar file with 1.8 GB, the uncompressed tar file of 30 GB and the fully expanded directory structure with 60 GB. It's 9.2m .hex files and 1.3m readme files. The readme files make some 12 GB of the github usage. Also, github does not preserve the hard links between identical bootloaders in the tarball, so on my own computer the 9.2m bootloaders only have a disk usage of 18 GB

Just released version u7.7 of the .hex repository and the urboot source - closing this issue

stefanrueger / urboot.hex

New directory structure and file name convention #3