search

platformio / platform-atmelavr

Atmel AVR: development platform for PlatformIO
https://registry.platformio.org/platforms/platformio/atmelavr
Apache License 2.0
138 stars 105 forks source link

Extra script for setting fuses and burning bootloader #153

Closed MCUdude closed 4 years ago

MCUdude commented 5 years ago

Hi!

As a developer working with AVRs I feel like the last thing missing from PlatformIO now is the ability to set the correct fuse bits and burn the correct bootloader based on the information given in platformio.ini.

MightyCore, MiniCore, and MegaCore (all supported by PlatformIO) comes with a bunch of pre-compiled bootloaders where all shares the same file names rules.

As long as board, board_build.f_cpu, and board_upload.speed is present in platformio.ini Is it possible for a script to figure out the correct fuse bits and load the correct bootloader hex file.

Preferably there should be three flags, one to only set the fuse bits, one to only burn the bootloader and one for doing both. Since the name and path of the bootloader are so extensive, we can check if the user entered a valid board_build.f_cpu, and board_upload.speed by checking if a bootloader with these values exists.

A bootloader for a >=64kB device is named like this:

bootloaders/atmega1284p/16000000L/optiboot_flash_atmega1284p_UART0_1000000_16000000L_BIGBOOT.hex

A bootloader for a <=32kB device is named like this:

bootloaders/atmega328p/16000000L/optiboot_flash_atmega328p_UART0_115200_16000000L.hex

By adding a new parameter to platformio.ini (board_upload.uart_port maybe?) we can specify if UART0, UART1 or UARTn should be used for uploading. Again, this decides what bootloader hex file is loaded.

Another fuse related feature is the ability to set the brownout detection level. This is possible in Arduino IDE through a separate menu option. Something like board_fuses.bod = 2.7V?

The last option should be to manually set fuses and override everything. My idea here would be to check if board_fuses.high, board_fuses.low and board_fuses.extended is present. If one or more is present, then this fuse is overridden by whatever value the particular boards_fuses fields holds

I have very little experience with python, but I think this should be doable, even for me if someone points me in the right direction. I also haven't worked with PlatformIO advanced scripting before, so forgive me if some of the following questions are silly to some.

EDIT: It seems like boards_mightycore.txt boards_minicore.txt and boards_megacore.txt already exist in my .platformio folder. Maybe these files can be parsed to make this job a little easier?

ivankravets commented 5 years ago

Hi! It should not be difficult. The right place is extending this target https://github.com/platformio/platform-atmelavr/blob/develop/builder/main.py#L247.

P.S: I added "Configuration" section to http://docs.platformio.org/en/latest/frameworks/arduino.html and references to your manuals on how to use PlatformIO + Cores.

MCUdude commented 5 years ago

P.S: I added "Configuration" section to http://docs.platformio.org/en/latest/frameworks/arduino.html and references to your manuals on how to use PlatformIO + Cores.

Awesome! Now it should be fairly easy for the average user to use my cores through PlatformIO 👍

It should not be difficult. The right place is extending this target https://github.com/platformio/platform-atmelavr/blob/develop/builder/main.py#L247.

OK, so this wouldn't be implemented in a separate script, but rather the main build script instead? So the commands would instead be pio target_fuses and pio target_bootloader? I want these commands to be available if board,board_build.f_cpu, and board_upload.speedis present. If they are, then we can start parsing the approperiate boards.txt files.

ivankravets commented 5 years ago

I think 2 targets are enough:

MCUdude commented 5 years ago

I think 2 targets are enough: fuses, which means setting fuses bootloader, which means burn bootloader?

Exactly. But we're not able to set the correct fuses and burn the correct bootloader file if we're missing information. Here's the idea:

Option 1: Only boards are present

defaults to board_build.f_cpu=16000000L defaults to board_upload.speed=115200 defaults to board_upload.uart_port=UART0 defaults to board_fuses.bod=2.7V

Option 2: boards,board_build.f_cpuand board_upload.speed are present

defaults to board_upload.uart_port=UART0 defaults to board_fuses.bod=2.7V Will not execute if bootloader file isn't found

Option 3: boards,board_build.f_cpu, board_upload.speed and board_upload.uart_portare present

defaults to board_fuses.bod=2.7V Will not execute if bootloader file isn't found

Option 4: boards,board_build.f_cpu, board_upload.speed and board_fuses.bodare present

defaults to board_upload.uart_port=UART0 Will not execute if bootloader file isn't found

Option 5: boards,board_build.f_cpu, board_upload.speed, board_fuses.bod and board_upload.uart_portare present

Will not execute if bootloader file isn't found

MCUdude commented 5 years ago

Maybe it's better if we're able to move everything fuse related into the manifest JSON files? Then there's no dependencies other then the bootloader files.

ivankravets commented 5 years ago

We did this for some boards. Do you mean this https://github.com/platformio/platform-atmelavr/blob/develop/boards/uno.json#L31 ?

MCUdude commented 5 years ago

Yes, but those fuses are fixed at BOD=2.7V, and external oscillator. What I'd like to implement is something that somehow generates the correct fuse bits based on what platformio.ini contains. If you look at the boards.txt files for MightyCore for instance, this is solved by representing the fuses in binary. Because PlatformIO now has separate manifest files for every target (e.g one for 324A, 324P, and 324PB) we only need binary representation for those that don't have an extended fuse.

If you want I can come up with a better pseudo-code example to show you better what I mean. But that would be later today because right now I'm on my way to the beach! 🏖 Anyways I think this would be a valuable and useful contribution to the PlatformIO project if I succeed!

ivankravets commented 5 years ago

If you can help with PR, it would be great! I hope you see how we are busy with generic PlatformIO Core, our IDE extensions. The time is limited for dev/platforms now :(

But that would be later today because right now I'm on my way to the beach! 🏖

He-he :) Have a nice relaxing!

MCUdude commented 5 years ago

If you can help with PR, it would be great!

I'll see what I can do. It will take some time though. And I'll probably ask some questions on the way if that's fine?

I hope you see how we are busy with generic PlatformIO Core, our IDE extensions. The time is limited for dev/platforms now :(

It's fully understandable, and I do not expect you do do this for me at all. It's just that I might need some input from you to have an OK starting point.

So for reference, how can I set fuses for the uno target? pio run --target fuses doesn't seem to work.

ivankravets commented 5 years ago

There is a new task in PlatformIO IDE for VSCode. Or, you can use PlatformIO CLI: “$ pio run -t fuses

MCUdude commented 5 years ago

I've been busy doing other things recently, but I just started looking at this again. To start out simple with a proof of concept I decided to stick with a separate extra_script for now. HOwever, I can't really get it to parse anything from platformio.ini.

Here's my platformio.ini file:

[env:env_custom_target]
platform = atmelavr
board = ATmega324P
framework = arduino

extra_scripts = extra_script.py
custom_message = "Hello World!"

And here's the script I try to run using this command: pio run --target testscript

try:
    import configparser
except ImportError:
    import ConfigParser as configparser

Import("env")

config = configparser.ConfigParser()
config.read("platformio.ini")
host = config.get("env_custom_target", "custom_message")

def mytarget_callback(*args, **kwargs):
    print("Hello PlatformIO!")
    print(host)

env.AlwaysBuild(env.Alias("testscript", None, mytarget_callback))

And here's the error I get:

NoSectionError: No section: 'env_custom_target':
  File "C:\Users\h.bull.LAUDM\.platformio\penv\lib\site-packages\platformio\builder\main.py", line 135:
    env.SConscript(item, exports="env")
  File "U:\.platformio\packages\tool-scons\script\..\engine\SCons\Script\SConscript.py", line 541:
    return _SConscript(self.fs, *files, **subst_kw)
  File "U:\.platformio\packages\tool-scons\script\..\engine\SCons\Script\SConscript.py", line 250:
    exec _file_ in call_stack[-1].globals
  File "C:\Users\h.bull.LAUDM\Documents\PlatformIO\Projects\ADV_SCRIPT\extra_script.py", line 10:
    host = config.get("env_custom_target", "custom_message")
  File "C:\Users\h.bull.LAUDM\.platformio\python27\Lib\ConfigParser.py", line 607:
    raise NoSectionError(section)

Could you point me in the right direction? It would be great to have a foundation to build on that just works.

ivankravets commented 5 years ago

See docs for ConfigParser https://docs.python.org/2/library/configparser.html

You need to do checking before .get(..). For example,

if config. has_section("..."):
    # do something
MCUdude commented 5 years ago

Thanks! However, it turned out that the section I had entered was incorrect. It was supposed to be env:env_custom_target instead of env_custom_target.

Not I have something I can work with. I'm a complete Python noob, as has to Google for even the simplest things. But that's my problem 😉

For a large project, the platformio.ini file may contain several environments or [env:myEnv]. Let's say one is [env:atmega324p] and the other one is [env:atmega1284p]. How can I invoke the script to handle only one of them, and not both, without physically modifying the script to match the current project? Somehow the environment I want to use has to somehow be passed to the script.

Something like this? pio run --target testscript -[someflag] env:atmega324p

or: pio run --target testscript env:atmega324p

ivankravets commented 5 years ago

If you need to get the only option from a current environment, just use this shortcut: https://github.com/platformio/platformio-core/blob/develop/platformio/builder/tools/pioproject.py#L28

print(env.GetProjectOption("my_option"))

In this case, no need to use ConfigParser at all.

MCUdude commented 5 years ago

Cool! Is it also possible to check if an option is present? This way I can make some options optional, like uart and bod. As you already know, I want the script to be able to figure out the correct fuse bits and later set fuse bits + burn bootloader based on the contents in the platformio.ini file.

But let's say I have these two environments. I only want to run the extra_script for the first one, not the second.

[env:env_custom_target]
platform = atmelavr
board = ATmega324P
framework = arduino

extra_scripts = extra_script.py

[env:env_custom_target2]
platform = atmelavr
board = ATmega324PA
framework = arduino
Import("env")

def fuses(*args, **kwargs):
    print(env.GetProjectOption("board"))

env.AlwaysBuild(env.Alias("testscript", None, fuses))

when I run pio run --target testscript I finally get this output.

Environment         Status    Duration
------------------  --------  ------------
env_custom_target   SUCCESS   00:00:12.942
env_custom_target2  FAILED    00:00:09.423

I don't want the script to run for the second environment. How can I prevent this?

ivankravets commented 5 years ago

Is it also possible to check if an option is present?

Just 

if env.GetProjectOption("option_exists"):
    # do something

# or

if not env.GetProjectOption("option_does_not_exists"):
    pass

I don't want the script to run for the second environment. How can I prevent this?

A lot of ways:

P.S: pio run -e env_custom_target -t testscript

MCUdude commented 5 years ago

Awesome! I'll see if I can implement this in a nice way.

At the moment I have a script that takes these parameters in the platformio.ini file:

board = ATmega1284P
board_build.f_cpu = 8000000L
hardware.oscillator = internal
hardware.bod = 2.7V
hardware.uart = uart0

And based on these parameters will calculate the correct lfuse, hfuse and efuse for all MightyCore, MiniCore, MegaCore, and MajorCore compatible microcontrollers. With all sub-variants (P, PA, PB etc) that's 31 different targets!

For a skilled Python programmer, the script is probably very ugly, but at least it gets the job done! Future plans are to add a way to burn the correct bootloader based on board, hardware.uart, board_build.f_cpu, and upload.speed.

MCUdude commented 5 years ago

OK, so I now have the fuses (lfuse, hfuse, efuse) stored in a variable each. How do I invoke Avrdude?

I need to build a command similar to this:

avrdude -Cavrdude.conf -v -p{board} -c{upload_protocol} -P{upload_flags} -e -Ulock:w:0x3f:m -Uefuse:w:{hfuse}:m -Uhfuse:w:{hfuse}:m -Ulfuse:w:{lfuse}:m
MCUdude commented 5 years ago

For reference, here's the script and ini file

[env:env_custom_target]
platform = atmelavr
board = ATmega32
framework = arduino
board_build.f_cpu = 1000000L
hardware.oscillator = internal
hardware.bod = disabled
hardware.uart = disabled
extra_scripts = extra_script.py
Import("env")

# Default values
target = str(env.GetProjectOption("board"))
f_cpu = "16000000L"
oscillator = "external"
bod = "2.7V"
uart = "uart0"

def get_lfuse():
    global target
    global f_cpu
    global oscillator
    global bod

    # Return manually defined lfuse if present in platformio.ini
    if(str(env.GetProjectOption("board_fuses.lfuse")) != "None"):
        return int(env.GetProjectOption("board_fuses.lfuse"), 0)

    if(target == "ATmega2561" or target == "ATmega2560"  or target == "ATmega1284"  or target == "ATmega1284P" or \
       target == "ATmega1281" or target == "ATmega1280"  or target == "ATmega644A"  or target == "ATmega644P"  or \
       target == "ATmega640"  or target == "ATmega328"   or target == "Atmega328P"  or target == "ATmega324A"  or \
       target == "ATmega324P" or target == "ATmega324PA" or target == "ATmega168"   or target == "ATmega168P"  or \
       target == "ATmega164A" or target == "ATmega164P"  or target == "ATmega88"    or target == "ATmega88P"   or \
       target == "ATmega48"   or target == "ATmega48P"):
        if(oscillator == "external"):
            return 0xf7
        else:
            if(f_cpu == "8000000L"):
                return 0xe2
            else:
                return 0x62

    elif(target == "ATmega328PB" or target == "ATmega324PB" or target == "ATmega168PB" or target == "ATmega162" or \
         target == "ATmega88PB"  or target == "ATmega48PB"  or target == "AT90CAN128"  or target == "AT90CAN64" or \
         target == "AT90CAN32"):
        if(oscillator == "external"):
            return 0xff
        else:
            if(f_cpu == "8000000L"):
                return 0xe2
            else:
                return 0x62

    elif(target == "ATmega8535" or target == "ATmega8515" or target == "ATmega32" or target == "ATmega16" or  \
         target == "ATmega8"):
        if(bod == "4.0V"):
            bod_bits = 0b00
        elif(bod == "2.7V"):
            bod_bits = 0b10
        else:
            bod_bits = 0b11

        if(oscillator == "external"):
            return (bod_bits << 6) + 0x3f
        else:
            if(f_cpu == "8000000L"):
                return (bod_bits << 6) + 0x24
            else:
                return (bod_bits << 6) + 0x21

    else:
        return -1

def get_hfuse():
    global target
    global uart
    global bod

    # Return manually defined hfuse if present in platformio.ini
    if(str(env.GetProjectOption("board_fuses.hfuse")) != "None"):
        return int(env.GetProjectOption("board_fuses.hfuse"), 0)

    if(target == "ATmega2561" or target == "ATmega2560" or target == "ATmega1284"  or target == "ATmega1284P" or \
       target == "ATmega1281" or target == "ATmega1280" or target == "ATmega644A"  or target == "ATmega644P"  or \
       target == "ATmega640"  or target == "ATmega328"  or target == "Atmega328P"  or target == "Atmega328PB" or \
       target == "ATmega324A" or target == "ATmega324P" or target == "ATmega324PA" or target == "ATmega324PB" or \
       target == "AT90CAN128" or target == "AT90CAN64"  or target == "AT90CAN32"):
        if(uart == "no_bootloader"):
            return 0xd7
        else:
            return 0xd6

    elif(target == "ATmega164A"  or target == "ATmega164P" or target == "ATmega162"):
        if(uart == "no_bootloader"):
            return 0xd5
        else:
            return 0xd4

    elif(target == "ATmega168" or target == "ATmega168P" or target == "ATmega168PB" or target == "ATmega88"  or \
         target == "ATmega88P" or target == "ATmega88PB" or target == "ATmega48"    or target == "ATmega48P" or \
         target == "ATmega48PB"):
        if(bod == "4.3V"):
            return 0xd4
        elif(bod == "2.7V"):
            return 0xd5
        elif(bod == "1.8V"):
            return 0xd6
        else:
            return 0xd7

    elif(target == "ATmega128" or target == "ATmega64" or target == "ATmega32"):
        if(oscillator == "external"):
            ckopt_bit = 0
        else:
            ckopt_bit = 1
        if(uart == "no_bootloader"):
            return 0xc7 + (ckopt_bit << 4)
        else:
            return 0xc6 + (ckopt_bit << 4)

    elif(target == "ATmega8535" or target == "ATmega8515" or target == "ATmega16" or target == "ATmega8"):
        if(oscillator == "external"):
            ckopt_bit = 0
        else:
            ckopt_bit = 1
        if(uart == "no_bootloader"):
            return 0xc5 + (ckopt_bit << 4)
        else:
            return 0xc4 + (ckopt_bit << 4)

    else:
        return -1

def get_efuse():
    global target
    global bod

    # Return manually defined efuse if present in platformio.ini
    if(str(env.GetProjectOption("board_fuses.efuse")) != "None"):
        return int(env.GetProjectOption("board_fuses.efuse"), 0)

    if(target == "ATmega2561" or target == "ATmega2560"  or target == "ATmega1284" or target == "ATmega1284P" or \
       target == "ATmega1281" or target == "ATmega1280"  or target == "ATmega644A" or target == "ATmega644P"  or \
       target == "ATmega640"  or target == "ATmega328"   or target == "Atmega328P" or target == "ATmega324A"  or \
       target == "ATmega324P" or target == "ATmega324PA" or target == "ATmega164A" or target == "ATmega164P"):
        if(bod == "4.3V"):
            return 0xfc
        elif(bod == "2.7V"):
            return 0xfd
        elif(bod == "1.8V"):
            return 0xfe
        else:
            return 0xff

    elif(target == "ATmega328PB" or target == "ATmega324PB"):
        if(bod == "4.3V"):
            return 0xf4
        elif(bod == "2.7V"):
            return 0xf5
        elif(bod == "1.8V"):
            return 0xf6
        else:
            return 0xf7

    elif(target == "ATmega168" or target == "ATmega168P" or target == "ATmega168PB" or target == "ATmega88" or \
         target == "ATmega88P" or target == "ATmega88PB"):
        if(uart == "no_bootloader"):
            return 0xfd
        else:
            return 0xfc

    elif(target == "ATmega128" or target == "ATmega64" or target == "ATmega48" or target == "ATmega48P"):
        return 0xff

    elif(target == "AT90CAN128" or target == "AT90CAN64" or target == "AT90CAN32"):
        if(bod == "4.1V"):
            return 0xfd
        elif(bod == "4.0V"):
            return 0xfb
        elif(bod == "3.9V"):
            return 0xf9
        elif(bod == "3.8V"):
            return 0xf7
        elif(bod == "2.7V"):
            return 0xf5
        elif(bod == "2.6V"):
            return 0xf3
        elif(bod == "2.5V"):
            return 0xf1
        else:
            return 0xff

    else:
        return -1

def fuses(*args, **kwargs):
    print("\n")
    global target
    global f_cpu
    global oscillator
    global bod
    global uart

    # Define F_CPU
    if(str(env.GetProjectOption("board_build.f_cpu")) != "None"):
        f_cpu = str(env.GetProjectOption("board_build.f_cpu"))
        print("Clock speed specified\t\tUsing board_build.f_cpu = %s" % f_cpu)
    else:
        print("Clock speed not specified\tUsing board_build.f_cpu = %s" % f_cpu)

    # Define internal or external oscillator
    if(str(env.GetProjectOption("hardware.oscillator")) == "internal" or str(env.GetProjectOption("hardware.oscillator")) == "external"):
        oscillator = str(env.GetProjectOption("hardware.oscillator"))
        print("Oscillator specified\t\tUsing hardware.oscillator = %s" % oscillator)
    else:
        print("Oscillator not specified\tUsing hardware.oscillator = %s" % oscillator)

    # Define BOD level
    if(str(env.GetProjectOption("hardware.bod")) != "None"):
        bod = str(env.GetProjectOption("hardware.bod"))
        print("BOD level specified\t\tUsing hardware.bod = %s" % bod)
    else:
        print("BOD level not specified\t\tUsing hardware.bod = %s" % bod)

    # Define UART port
    if(str(env.GetProjectOption("hardware.uart")) == "uart0" or str(env.GetProjectOption("hardware.uart")) == "uart1" or str(env.GetProjectOption("hardware.uart")) == "uart2" or str(env.GetProjectOption("hardware.uart")) == "uart3"):
        uart = str(env.GetProjectOption("hardware.uart"))
        print("UART port specified\t\tUsing hardware.uart = %s" % uart)
    elif(str(env.GetProjectOption("hardware.uart")) != "None"):
        uart = "no_bootloader"
        print("UART not specified\t\tNo bootloader will be installed")
    else:
        print("UART port not specified\t\tDefault is hardware.uart = %s" %uart)

    low_fuse = hex(get_lfuse())
    high_fuse = hex(get_hfuse())
    ext_fuse = hex(get_efuse())

    print("\nCalculated low fuse:  %s" % low_fuse)
    print("Calculated high fuse: %s" % high_fuse)
    print("Calculated ext fuse:  %s" % ext_fuse)

    # Invoke Avrdude here!

env.AlwaysBuild(env.Alias("fusess", None, fuses))
MCUdude commented 4 years ago

@ivankravets sorry for bothering you, but I'm just not skilled enough to figure out how to invoke Avrdude at this point., so I'm pretty much stuck 😞

ivankravets commented 4 years ago

We would like to have this as a part of dev/platform. Is it ok for you? I think you provided a lot of valuable information. I propose to move this code to builder/bootloader.py and extend with bootloader target.

Could you post here a few commands how should look AVRdude command? How many of them we should call per 1 bootloader flashing?

MCUdude commented 4 years ago

That would be great! What's important to me is that it's easy to use this functionality

I think it would be great if we had separate commands for setting fuses and burning bootloader. For small ATtiny AVRs, a bootloader doesn't make sense, but fuses are still very important.

Here's a suggestion on how the fuses command could/should look like: pio target fuses. This would invoke the following Avrdude command:

avrdude -Cavrdude.conf -v -p{board} -c{upload_protocol} -P{upload_flags} -Ulock:w:0x3f:m -Uefuse:w:{hfuse}:m -Uhfuse:w:{hfuse}:m -Ulfuse:w:{lfuse}:m

Here's a suggestion on how the bootloader command could/should look like: pio target bootloader. This would invoke the following Avrdude command:

avrdude -Cavrdude.conf -v -p{board_build.mcu} -c{upload_protocol} -P{upload_flags -e -Uflash:w:/path/to/bootloaders/optiboot_flash/bootloaders/{board_build.mcu}/{board_build.f_cpu}/optiboot_flash_board_build.mcu_{hardware.uart}_{board_upload.speed}_{board_build.f_cpu}.hex:i -Ulock:w:0x0f:m

Note that some chips like ATmega8515, ATmega8535, ATmega8, ATmega16, and ATmega32 doesn't have an efuse. In order to not have a special fuses command just for these, we can add a "fake" efuse option in the avrdude.conf file. I've done this with MightyCore's avrdude.conf for instance.

Have a look at my Optiboot flash repo on how different hex files for different targets are handled. This repo covers pretty much all AVR based boards out there, except the official Arduino Mega 2560 (uses an stk500v2 bootloader instead) and Arduino Nano (uses ATMEGABOOT instead).

If this makes into the official PlatformIO core it would make a BIG difference for AVR and Arduino developers who are working bare hardware and not just development boards. This is exciting!

EDIT: Would it then make sense to provide two separate scripts, builder/fuses.py and builder/bootloader.py?

MCUdude commented 4 years ago

Another important feature would be to manually override the generated fuses. If you have a very specific need for a certain type of fuse setting, it should be possible to specify it in platformio.ini, like this. I can easily add this to the original script!

board_fuses.hfuse = 0x[nn]
board_fuses.lfuse = 0x[nn]
board_fuses.efuse = 0x[nn]
ivankravets commented 4 years ago

@valeros will back soon here with the updates.

MCUdude commented 4 years ago

Great! Looking forward to discussing these features with you. BTW I've updated the script I posted earlier. Not it supports overriding of lfuse, hfuse, and efuse in platformio.ini.

MCUdude commented 4 years ago

OK, I've done some more work on the fuses script.

@valeros when will I hear from you?

Import("env")

# Default values
target = str(env.GetProjectOption("board"))
f_cpu = "16000000L"
oscillator = "external"
bod = "2.7v"
eesave = "yes"
uart = "uart0"

def get_lfuse():
    global target
    global f_cpu
    global oscillator
    global bod
    global eesave

    # Return manually defined lfuse if present in platformio.ini
    if(str(env.GetProjectOption("board_fuses.lfuse")) != "None"):
        return int(env.GetProjectOption("board_fuses.lfuse"), 0)

    if(target == "ATmega2561" or target == "ATmega2560"  or target == "ATmega1284"  or target == "ATmega1284P" or \
       target == "ATmega1281" or target == "ATmega1280"  or target == "ATmega644A"  or target == "ATmega644P"  or \
       target == "ATmega640"  or target == "ATmega328"   or target == "Atmega328P"  or target == "ATmega324A"  or \
       target == "ATmega324P" or target == "ATmega324PA" or target == "ATmega168"   or target == "ATmega168P"  or \
       target == "ATmega164A" or target == "ATmega164P"  or target == "ATmega88"    or target == "ATmega88P"   or \
       target == "ATmega48"   or target == "ATmega48P"):
        if(oscillator == "external"):
            return 0xf7
        else:
            if(f_cpu == "8000000L"):
                return 0xe2
            else:
                return 0x62

    elif(target == "ATmega328PB" or target == "ATmega324PB" or target == "ATmega168PB" or target == "ATmega162" or \
         target == "ATmega88PB"  or target == "ATmega48PB"  or target == "AT90CAN128"  or target == "AT90CAN64" or \
         target == "AT90CAN32"):
        if(oscillator == "external"):
            return 0xff
        else:
            if(f_cpu == "8000000L"):
                return 0xe2
            else:
                return 0x62

    elif(target == "ATmega8535" or target == "ATmega8515" or target == "ATmega32" or target == "ATmega16" or  \
         target == "ATmega8"):
        if(bod == "4.0v"):
            bod_bits = 0b11
        elif(bod == "2.7v"):
            bod_bits = 0b01
        else:
            bod_bits = 0b00

        if(oscillator == "external"):
            return 0xff & ~(bod_bits << 6)
        else:
            if(f_cpu == "8000000L"):
                return 0xe4 & ~(bod_bits << 6)
            else:
                return 0xe1 & ~(bod_bits << 6)

    elif(target == "ATtiny13" or target == "ATtiny13A"):
        # Get eesave value
        if(eesave == "yes"):
            eesave_bit = 1
        else:
            eesave_bit = 0
        if(oscillator == "external"):
            return 0x78 & ~(eesave_bit << 6)
        else:
            if(f_cpu == "9600000L"):
                return 0x7a & ~(eesave_bit << 6)
            elif(f_cpu == "4800000L"):
                return 0x79 & ~(eesave_bit << 6)
            elif(f_cpu == "1200000L"):
                return 0x6a & ~(eesave_bit << 6)
            elif(f_cpu == "600000L"):
                return 0x69 & ~(eesave_bit << 6)
            elif(f_cpu == "128000L"):
                return 0x7b & ~(eesave_bit << 6)
            elif(f_cpu == "16000L"):
                return 0x6b & ~(eesave_bit << 6)

    else:
        return -1

def get_hfuse():
    global eesave
    global oscillator
    global target
    global uart
    global bod

    # Return manually defined hfuse if present in platformio.ini
    if(str(env.GetProjectOption("board_fuses.hfuse")) != "None"):
        return int(env.GetProjectOption("board_fuses.hfuse"), 0)

    # Get eesave value
    if(eesave == "yes"):
        eesave_bit = 1
    else:
        eesave_bit = 0

    # Get ckopt for targets that uses this
    if(oscillator == "external"):
        ckopt_bit = 1
    else:
        ckopt_bit = 0

    if(target == "ATmega2561" or target == "ATmega2560" or target == "ATmega1284"  or target == "ATmega1284P" or \
       target == "ATmega1281" or target == "ATmega1280" or target == "ATmega644A"  or target == "ATmega644P"  or \
       target == "ATmega640"  or target == "ATmega328"  or target == "Atmega328P"  or target == "Atmega328PB" or \
       target == "ATmega324A" or target == "ATmega324P" or target == "ATmega324PA" or target == "ATmega324PB" or \
       target == "AT90CAN128" or target == "AT90CAN64"  or target == "AT90CAN32"):
        if(uart == "no_bootloader"):
            return 0xdf & ~(eesave_bit << 3)
        else:
            return 0xde & ~(eesave_bit << 3)

    elif(target == "ATmega164A"  or target == "ATmega164P" or target == "ATmega162"):
        if(uart == "no_bootloader"):
            return 0xdd & ~(eesave_bit << 3)
        else:
            return 0xdc & ~(eesave_bit << 3)

    elif(target == "ATmega168" or target == "ATmega168P" or target == "ATmega168PB" or target == "ATmega88"  or \
         target == "ATmega88P" or target == "ATmega88PB" or target == "ATmega48"    or target == "ATmega48P" or \
         target == "ATmega48PB"):
        if(bod == "4.3v"):
            return 0xdc & ~(eesave_bit << 3)
        elif(bod == "2.7v"):
            return 0xdd & ~(eesave_bit << 3)
        elif(bod == "1.8v"):
            return 0xde & ~(eesave_bit << 3)
        else:
            return 0xdf & ~(eesave_bit << 3)

    elif(target == "ATmega128" or target == "ATmega64" or target == "ATmega32"):
        if(uart == "no_bootloader"):
            return 0xdf & ~(ckopt_bit << 4) & ~(eesave_bit << 3)
        else:
            return 0xde & ~(ckopt_bit << 4) & ~(eesave_bit << 3)

    elif(target == "ATmega8535" or target == "ATmega8515" or target == "ATmega16" or target == "ATmega8"):
        if(uart == "no_bootloader"):
            return 0xdd & ~(ckopt_bit << 4) & ~(eesave_bit << 3)
        else:
            return 0xdc & ~(ckopt_bit << 4) & ~(eesave_bit << 3)

    elif(target == "ATtiny13" or target == "ATtiny13A"):
        if(bod == "4.3v"):
            return 0x9
        elif(bod == "2.7v"):
            return 0xfb
        elif(bod == "1.8v"):
            return 0xfd
        else:
            return 0xff

    else:
        return -1

def get_efuse():
    global target
    global bod

    # Return manually defined efuse if present in platformio.ini
    if(str(env.GetProjectOption("board_fuses.efuse")) != "None"):
        return int(env.GetProjectOption("board_fuses.efuse"), 0)

    if(target == "ATmega2561" or target == "ATmega2560"  or target == "ATmega1284" or target == "ATmega1284P" or \
       target == "ATmega1281" or target == "ATmega1280"  or target == "ATmega644A" or target == "ATmega644P"  or \
       target == "ATmega640"  or target == "ATmega328"   or target == "Atmega328P" or target == "ATmega324A"  or \
       target == "ATmega324P" or target == "ATmega324PA" or target == "ATmega164A" or target == "ATmega164P"):
        if(bod == "4.3v"):
            return 0xfc
        elif(bod == "2.7v"):
            return 0xfd
        elif(bod == "1.8v"):
            return 0xfe
        else:
            return 0xff

    elif(target == "ATmega328PB" or target == "ATmega324PB"):
        if(bod == "4.3v"):
            return 0xf4
        elif(bod == "2.7v"):
            return 0xf5
        elif(bod == "1.8v"):
            return 0xf6
        else:
            return 0xf7

    elif(target == "ATmega168" or target == "ATmega168P" or target == "ATmega168PB" or target == "ATmega88" or \
         target == "ATmega88P" or target == "ATmega88PB"):
        if(uart == "no_bootloader"):
            return 0xfd
        else:
            return 0xfc

    elif(target == "ATmega128" or target == "ATmega64" or target == "ATmega48" or target == "ATmega48P"):
        return 0xff

    elif(target == "AT90CAN128" or target == "AT90CAN64" or target == "AT90CAN32"):
        if(bod == "4.1v"):
            return 0xfd
        elif(bod == "4.0v"):
            return 0xfb
        elif(bod == "3.9v"):
            return 0xf9
        elif(bod == "3.8v"):
            return 0xf7
        elif(bod == "2.7v"):
            return 0xf5
        elif(bod == "2.6v"):
            return 0xf3
        elif(bod == "2.5v"):
            return 0xf1
        else:
            return 0xff

    else:
        return -1

def fuses(*args, **kwargs):
    print("\n")
    global target
    global f_cpu
    global oscillator
    global bod
    global eesave
    global uart

    # Define F_CPU
    if(str(env.GetProjectOption("board_build.f_cpu")) != "None"):
        f_cpu = str(env.GetProjectOption("board_build.f_cpu")).upper()
        print("Clock speed specified\t\tUsing board_build.f_cpu = %s" % f_cpu)
    else:
        print("Clock speed not specified\tUsing board_build.f_cpu = %s" % f_cpu)

    # Define internal or external oscillator
    if(str(env.GetProjectOption("hardware.oscillator")).lower() == "internal" or str(env.GetProjectOption("hardware.oscillator")).lower() == "external"):
        oscillator = str(env.GetProjectOption("hardware.oscillator")).lower()
        print("Oscillator specified\t\tUsing hardware.oscillator = %s" % oscillator)
    else:
        print("Oscillator not specified\tUsing hardware.oscillator = %s" % oscillator)

    # Define BOD level
    if(str(env.GetProjectOption("hardware.bod")) != "None"):
        bod = str(env.GetProjectOption("hardware.bod")).lower()
        print("BOD level specified\t\tUsing hardware.bod = %s" % bod)
    else:
        print("BOD level not specified\t\tUsing hardware.bod = %s" % bod)

    # Define EE save
    if(str(env.GetProjectOption("hardware.eesave")).lower() == "true" or str(env.GetProjectOption("hardware.eesave")).lower() == "yes" or str(env.GetProjectOption("hardware.eesave")) == "enabled"):
        eesave = "yes"
        print("EESAVE specified\t\tEEPROM will be retained")
    elif(str(env.GetProjectOption("hardware.eesave")).lower() == "false" or str(env.GetProjectOption("hardware.eesave")).lower() == "no" or str(env.GetProjectOption("hardware.eesave")) == "disabled"):
        eesave = "no"
        print("EESAVE specified\t\tEEPROM will not be retained")
    else:
        eesave = "yes"
        print("EESAVE not specified\t\tEEPROM will be retained")

    # Define UART port
    if(str(env.GetProjectOption("hardware.uart")).lower() == "uart0" or str(env.GetProjectOption("hardware.uart")).lower() == "uart1" or str(env.GetProjectOption("hardware.uart")).lower() == "uart2" or str(env.GetProjectOption("hardware.uart")).lower() == "uart3"):
        uart = str(env.GetProjectOption("hardware.uart")).lower()
        print("UART port specified\t\tUsing hardware.uart = %s" % uart)
    elif(str(env.GetProjectOption("hardware.uart")) != "None"):
        uart = "no_bootloader"
        print("UART not specified\t\tNo bootloader will be installed")
    else:
        print("UART port not specified\t\tDefault is hardware.uart = %s" %uart)

    low_fuse = hex(get_lfuse())
    high_fuse = hex(get_hfuse())
    ext_fuse = hex(get_efuse())

    print("\nCalculated low fuse:  %s" % low_fuse)
    print("Calculated high fuse: %s" % high_fuse)
    print("Calculated ext fuse:  %s" % ext_fuse)

    # Invoke Avrdude here!

env.AlwaysBuild(env.Alias("fuses", None, fuses))
MCUdude commented 4 years ago

@ivankravets would you like to show me how an Avrdude command in this script would look like? While I have everything fresh in memory I'd like to continue working on this script while I'm waiting for @valeros to respond. The next step would be to add an Avrdude command to fuses() and start working on the bootloader part.

ivankravets commented 4 years ago

Sorry for the delay :( @valeros is a little bit busy with something sweet 💣 for PlatformIO Core 😊

He will back soon. I think this part of code should help https://github.com/platformio/platform-atmelavr/blob/develop/builder/main.py#L191:L212

MCUdude commented 4 years ago

I think this part of code should help https://github.com/platformio/platform-atmelavr/blob/develop/builder/main.py#L191:L212

I've tried this already but wasn't able to figure it out. Could you provide a somehow stripped-down version of this I can experiment with?

ivankravets commented 4 years ago

Invoke Avrdude here!

env.Execute("avrdude --version")
MCUdude commented 4 years ago

Nice, that works. Last question (for now 😉); how to I refer to the avrdude.conf file PlatformIO comes with? Obviously this won't work:

env.Execute("avrdude -C avrdude.conf -cusbasp")
ivankravets commented 4 years ago 
import os

platform = env.PioPlatform()
avrdude_dir = platform.get_package_dir("tool-avrdude")
avrdude_conf = os.path.join(avrdude_dir, "avrdude.conf")
cmd = [
  "-p", "$BOARD_MCU", 
  "-C", avrdude_conf
]
env.Execute(cmd)

P.S: I've not tested it :)

MCUdude commented 4 years ago

It didn't work directly, but you provided enough information for me to figure it out.

Do you know why I'm getting single quotes and square brackets around the string when I'm trying to get upload_flags from platformio.ini?

if upload_flags = -Pusb is defined like this in platformio.ini, then str(env.GetProjectOption("upload_flags")) will return ['-Pusb'], not -Pusb like I'm used to.

ivankravets commented 4 years ago

env.GetProjectOption("upload_flags") - this is array. You can directly extend it with cmd.

cmd.extend(env.GetProjectOption("upload_flags", []))
ivankravets commented 4 years ago

Maybe this will help

print(" ".join(env.GetProjectOption("upload_flags", [])))
MCUdude commented 4 years ago

@ivankravets thanks! With a little tweaking, I now have a functional script that will load the correct fuses to (almost) any AVR target based on parameters specified in platformio.ini. I decided to create a temporary repo where I host the script/project. I figured this would be easier for all of us. It's hosted here: https://github.com/MCUdude/pio-script

The next task would be to work on the bootloader command: pio run --target bootloader. I will be using Optiboot flash since it is tested and proved, and is practically identical to the official Optiboot code. The strict naming of the bootloader hex files makes it really easy to get the correct bootloader file based on hardware.uart, board_build.f_cpu and board_upload.speed.

ivankravets commented 4 years ago

Great! You are now supe experienced Python dev 😊

I see that Arduino Core keeps FUSES in boards.txt in bootloader.*** section. Does it make sense for us to pre-fill boards/***.json with fuses data? Or, your dynamic way is better?

MCUdude commented 4 years ago

Great! You are now super experienced Python dev 😊

Haha, not at all! 😁 I feel like I'm writing a C program with a weird syntax. But hey, it does work!

I see that Arduino Core keeps FUSES in boards.txt in bootloader.*** section

bootloader. is to match the platform.txt file. The fuses are actually loaded as a separate command before the actual bootloader, so it would actually make sense to rename it to fuses. in boards.txt and platform.txt. But it's just for looks, nothing critical.

Does it make sense for us to pre-fill boards/***.json with fuses data? Or, your dynamic way is better?

If the script is watertight, there is no need to store any fuses in the json manifests. f nothing other than board is specified, the calculated fuses will be "safe" no matter what your hardware is. If the user totally screws up the configuration to a point where it doesn't even make sense the script should write fuses that don't lock the user out. If the script is ran for a target that isn't supported the script should not do anything other than writing an error message to the screen.

Here are some rules I've implemented:

The script still needs lots of testing, especially when the options in platformio.ini is randomly picked.

MCUdude commented 4 years ago

@ivankravets @valeros So what will happen next? The script is pretty much finished. I'm sure you python gurus can improve the overall syntax to make it more elegant and easier to extend with support for other targets as well (@SpenceKonde's ATTinyCore is the first that comes to mind).

I'm not sure where you plan to place the Optiboot flash folder, so I've placed it under /packages/framework-arduinoavr

liebman commented 4 years ago

Please also look at #157 - need to make sure that flash is not always erased when setting fuses. If one is setting the reset pin to an IO pin then erasing flash removes firmware loaded before and device is no longer programmable (without a HV programmer)....

MCUdude commented 4 years ago

The script I've made does not support RSTDISABLE as of today, to prevent users from messing up. As of today, the script is mostly ATmega related since I mostly work with ATmegas. Other fuse settings for ATtinys could/should be added in the future to make this script even more versatile.

But yeah, it's probably better to erase the flash memory when the bootloader is burned instead. I'll look into it.

@ivankravets I haven't heard from you in a week. Just busy or abandoned the idea of implementing this into the pio core?

ivankravets commented 4 years ago

@MCUdude It's in our TODO :)

valeros commented 4 years ago

Hi @MCUdude ! Many thanks for preparing such a good basis for this feature. I've created a special branch with the initial implementation based on the code snippet you posted above. So, most of the changes are stylistic, I also used the mcu field as the target (I hope that'll enable this feature for more boards). Also, would be great if you could give meaningful names for the sets of mcus used in calculations (Is there a reason why you grouped them in that way?). Feel free comment on any issues you can find.

MCUdude commented 4 years ago

Great clean up! I didn't use the mcu field because I wasn't able to retrieve it from the manifest files. Does this mean the MCU parameter has to be defined in the ini file for this to work? That shouldn't be necessary many manifest files, as their board name actually matches the chip type. maybe you can make the mcu field optional by checking if the board name is a valid target for this script? Just a thought.

Is there a reason why you grouped them in that way?

The reason why they're grouped the way they are under each fuse function is that their properties are identical, so the calculated value will be the same for the entire group. I used the Engbedded fusecalc a lot when I worked on this. Some chips have identical lfuse values, some have identical hfuse and some efuse. There isn't really any good patterns. It depends on flash size, chip families, "old" vs "new" AVRs (ATmega8 is old, ATmega88 is new for instance). Fuse-vise, the ATmega328P is for instance much closer to ATmega324A/P/PA than ATmega168/P, while Atmega168P/PA is not close to ATmega164A/P at all.

Bear in mind that ATmega8535, ATmega8515, ATmega8, ATmega16 and ATmega32 doesn't have an efuse. To me it looks like it terminates the script of efuse can't be calculated for.

Will you implement bootloader burning in a different script? Any thought about using Optiboot flash for this?

valeros commented 4 years ago

@MCUdude If there is no mcu field then we'll fall back to the board name, WDYT? Earlier you mentioned that the --t bootloader and --t fuses targets can be used separately, but Arduino IDE executes these two steps as a single target, first it sets fuses and then uploads the bootloader. Can we use the fuses calculated by your script and then upload bootloader or we need to use the fuses specified in boards.txt? At the moment, there are no bootloader binaries in our packages, so will it be OK if we make the field to the path to the bootloader a mandatory field in order to program bootloader? 

board_bootloader.path = bootloaders/optiboot_atmega328.hex
MCUdude commented 4 years ago

@MCUdude If there is no mcu field then we'll fall back to the board name, WDYT?

I assume we're talking about the contents of platformio.ini. It sounds like a great idea! if build_board.mcu is missing, the fall back to using board instead.

Earlier you mentioned that the --t bootloader and --t fuses targets can be used separately, but Arduino IDE executes these two steps as a single target, first it sets fuses and then uploads the bootloader. Can we use the fuses calculated by your script and then upload bootloader or we need to use the fuses specified in boards.txt?

You can safely use the fuses calculated in the script. If the fields are correct in platformio.ini, the fuses will take the bootloader space into account when calculating the fuses. The reason why I suggested adding this to two separate commands is that some users don't need a bootloader for their project, and some chips don't support having a bootloader. Having two separate commands for this gives the user greater flexibility. PlatformIO is supposed to be more advanced than Arduino IDE? Maybe we can add a third command to both set fuses and then burn bootloader?

At the moment, there are no bootloader binaries in our packages, so will it be OK if we make the field to the path to the bootloader a mandatory field in order to program bootloader?

I'm not sure this is a very good idea. What kind of bootloader that should be loaded highly depends on the configuration done by the user in platformio.ini. We have clock speed, baud rate and UART port for instance. It would IMO be much better if you included the optiboot_flash binaries instead because then the end-user doesn't have to care about what exact hex file is loaded. What's important is that the bootloader hex file matches the platformio.ini configuration.

valeros commented 4 years ago

Maybe we can add a third command to both set fuses and then burn bootloader?

Is there a reason why they set fuses before uploading the bootloader? Can we burn bootloader without setting the fuses?

You can safely use the fuses calculated in the script. If the fields are correct in platformio.ini, the fuses will take the bootloader space into account when calculating the fuses.

Is it OK if calculated fuses for uno board differ from the fuses specified in boards.txt?

What kind of bootloader that should be loaded highly depends on the configuration done by the user in platformio.ini.

So binaries in your repository are fully compatible with bootloaders for all boards shipped with Arduino IDE (e.g gemma, leonardo, etc) ? In other words, will we be able to program boards from the Arduino IDE after programming your precompiled bootloader binary?

MCUdude commented 4 years ago

Is there a reason why they set fuses before uploading the bootloader? Can we burn bootloader without setting the fuses?

Let's say you want to want to change the upload speed. You can do this by only re-burn the bootloader. No need to re-set the same fuses again. But it's not a very big deal. It's to just have one command. It's also less confusing which parameter changes require new fuses and which require a new bootloader.

So binaries in your repository are fully compatible with bootloaders for all boards shipped with Arduino IDE (e.g gemma, leonardo, etc) ? In other words, will we be able to program boards from the Arduino IDE after programming your precompiled bootloader binary?

No, the ATmega32U4 found on e.g Leonardo uses a completely different bootloader, and we can't play around with fuses the same way.

I have a suggestion though. Messing with fuses and various bootloaders isn't what you really need if you're working with a ready-made development board such as Arduino UNO. It's when you're working on custom hardware with a hack-picked microcontroller you need it. Maybe we could make this script only available to only some targets? Let's say you're playing around with an Arduino UNO. That's completely fine, and you select uno as your board. Let's now say you want to play with fuses and different upload speeds. Now you're technically not using the UNO as it was intended too, so instead you can switch out board = uno with board = ATmega328P. Now all "low-level" features are available. Is this a good idea? It's close to impossible to support custom fuse bits for all AVR-based boards out there, but we still can provide a suited version of optiboot + the correct fuse bits for many AVRs.

valeros commented 4 years ago

We'd like to allow users to program bootloaders even for development boards (at least users should be able to burn the bootloaders bundled with Arduino IDE). So to sum up:

Does this process sound good to you? Also, are values for unlock_bits and lock_bits the same for your cores? Thanks!

MCUdude commented 4 years ago

Sounds good to me! unlock_bits and lock_bits are the same for my cores as well.

A few other things:

To sum it up: So my cores (only) will support both the fuses and bootloader commands. And the correct Optiboot_flash hex file will be loaded based on the parameters from platformio.ini.

On "board" targets (uno, mega etc) the fuses command is not available. It is possible to burn bootloader, but this will load the default fuses and the default bootloader specified in the corresponding manifest file. Fuses will for now only be calculated if the target "belongs" to MightyCore, MiniCore, MegaCore (and perhaps MajorCore in the future?)