PROGMEM_SECTION Programming issues

[alxgarza]

When using PROGMEM_SECTION other than 0, I get verification errors.

Using PROGMEM_SECTION1: pymcuprog.programmer - ERROR - Verify failed: Verify mismatch starting at location 0x008000: 0x07 vs 0xFF Using PROGMEM_SECTION2: pymcuprog.programmer - ERROR - Verify failed: Verify mismatch starting at location 0x010000: 0x07 vs 0xFF Using PROGMEM_SECTION3: pymcuprog.programmer - ERROR - Verify failed: Verify mismatch starting at location 0x018000: 0x00 vs 0xFF

Using PROGMEM_SECTION0, I do not get any error. But I can't read the data stored.

But using PROGMEM I can read the data.

Any ideas? I'm using a AVR128DA and DxCode 1.4.10

[SpenceKonde]

This appears to be a bug in SerialUPDI? Not sure. No ETA on fix it is unlikely to be fixed in 1.5.0. I don't know if it's failing to read or failing to write the data. Either way, it's likely going to be a big fucking deal to fix, as it is symptomatic of SerialUPDI not dealing well with "holes" in the flash image.

You can only access data as if it were a normal variable if the section matches flmap setting, which defaults to 3 on 128k parts and 1 on 64k parts, since only 32k of flash can be mapped to the main address space. So that part is expected

[mosqu-ito]

I'm old, and my Arduino knowledge coming into this was old, Uno vintage. This report, along with the various warnings from Our Great Benefactor at https://github.com/SpenceKonde/DxCore/blob/master/megaavr/extras/Ref_PROGMEM.md scared me so badly that for a while I was going out of my way to buy only 32K parts so that "No special considerations are needed".

My younger self cared a lot about speed and efficiency. I can remember carefully tweaking assembler for the Zilog Z-80 and the Motorola 6809, I even tried overclocking some of the early Intel CPUs. But once you get to a certain age, you realize that some variant of Moore's Law is probably working faster in that direction than you are. Now, if I want something to run faster, I just go take a nap and then buy the newer, faster one.

So, with regard to the 128K parts, I knew I would be happy if I could just write bigger programs (yup, seems to work right out of the box) and store larger tables of constants in flash. I don't need the latter to be highly optimized for speed, I just want Updi to stuff it in there, and to be able to read it at runtime.

Well, I seem to be able to do that much. I've been playing with some AVR128DB28 chips. I dispense with the bootloader, I set the Flash Writing to Disabled (BootSize=0, CodeSize=0) because I'm happy to let Updi write the flash for me. I use simple "const PROGMEM" declarations and don't bother myself with where exactly they're going. And then, yes, I just use "pgm_get_far_address" and "pgm_read_byte_far".

I'm not suggesting that this is optimal, Our Great Benefactor has specifically stated otherwise, but it seems to be working, right up into the high addresses. "Sketch uses 121880 bytes (92%) of program storage space. Maximum is 131072 bytes." (mostly silly data I put up there just to check that I could indeed read it)

So... assuming I am content with my modest ambitions, is there any reason to believe that I am going to run into a roadblock, and suddenly discover that my current seemingly-successful approach is going to crash and burn?

If not, then I post this little blurb for those who perhaps, like me, have been nervous about buying the large-flash parts. For basic needs, they seem to work just fine right out of the box.

[SpenceKonde]

No, if you use the flash like you did one the old parts this will work, Quentin is in town (will be until march,) and I consider this a top priority bug in SerialUPDI

Note that the sections should work correctly if you use jtag2updi to upload. My current understanding is that SerialUPDI is not properly handling "holes" in the hex file,

[alxgarza]

I have somewhat tested what mosqu-ito comments. With progmem and blocks of about 8K each, I have been able to flash them and read them with no problems. No sections used.

Writing 68832 bytes of data to flash...

[SpenceKonde]

It's worth noting that yoju can put almost 64k of data into progmem before it gets into far addresses.

You only get 32k of mapped flash (so the flash looks like: (a small amount of stuff at the start) PROGMEM and then code until the end of flash. By default, the last 32k is mapped. so like [PM][PM][FAR][MAP] with each section being 32k - there's an easy way to read the first half of the flash like on a classic AVR, and there's an easy way to read the last quarter of the flash, it's theone in between that's awkward (we strongly recommend against changing flmap!) The main difference is that you can read it like any variable instead of with pgm_read_byte_far. Intrinsically, the instructions are equally fast: ld is 2 clocks + 1 if accessing nvm, lpm is 3 clocks and always accesses the flash. The mapped flash has two consequences in addition to being much cleaner looking:

1. Each pgm_read_byte is a separate and unconnected operation. With mapped flash (or sram) it will be able to notice that (say) oh you're reading 20 characters) and make a copy of the address, in a pointer register and read the 20 bytes using ld with postincrement = 60 clock cycles. With pgm_read_byte, it doesnt' know what pgm_read_byte does, so it can't do that. It instead just gets to use lpm. Then between each word it increments it with ADIW (2 clocks). So it take 80 clocks instead of 60.
2. lpm and elpm use the Z pointer only, so it may impose extra overhead in that way, plus it has to set the RAMPZ register (and will set this, needed or not every time.. Plus, it can't directly have the compiler ensure that the low two bytes of the pointer re in the z register so it needs a movw as well, So in this case it takes 140 clocks.

A nifty bonus of mapped flash if it's set to the default last 32k on a 128k chip is that the address (with the first bit truncated) of the far address is the same as the address when accessing it through the mapped progmem, hence you can just declare a variable in that section and it literally just works. the high bit is truncated, and that's the same as the address as the variable is in mapped flash.

The bug priority for SerialUPDI is 1) Fix issue with holes in flash 2) Improve quality of error messages 3) Add a means of locking the chip (we already got an auto-unlock mechansim via PR) 4) Add the "write userrow of a locked chip" thing,

[SpenceKonde]

Can I ask you a question about your issue? Can you confirm that it happens with serialUPDI?

[alxgarza]

I am running the python script prog.py located in: Arduino15/packages/DxCore/hardware/megaavr/1.4.10/tools/prog.py

[SpenceKonde]

Thanks! I will reprodce the bug and if i see it, I will notify quentin at one ofcthis critical release blocking issue for dxcore at once. Jtah2updi, if it works (it's unreliable as shit) it will handle ut correctly. Pretty sure it's his code that's busted.

---

Spence Konde Azzy’S Electronics

New products! Check them out at tindie.com/stores/DrAzzy GitHub: github.com/SpenceKonde ATTinyCore: Arduino support for almost every ATTiny microcontroller Contact: @.***

On Thu, Oct 20, 2022, 11:39 alxgarza @.***> wrote:

I am running the python script prog.py located in: Arduino15/packages/DxCore/hardware/megaavr/1.4.10/tools/prog.py

— Reply to this email directly, view it on GitHub https://github.com/SpenceKonde/DxCore/issues/344#issuecomment-1285770507, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTXEW6LR3GPWNIBTV5XIY3WEFRU3ANCNFSM6AAAAAAQJCWNCY . You are receiving this because you commented.Message ID: @.***>

[mosqu-ito]

Maybe it's not holes? I've seen Updi do multiple upload and verify passes as if it understands holes. Maybe the linker doesn't understand about your bit-15 trick and is combining sections? Maybe Updi isn't handling flash page boundaries correctly?

I did some experiments, I give you here the results in the hope that something might provide a clue to fuel your big noggin.

// _DOES_ verify...
//const char Sig0[256] PROGMEM_SECTION0 = "Section 0 Signature";
//const char Sig1[256] PROGMEM_SECTION1 = "Section 1 Signature";
//const char Sig2[256] PROGMEM_SECTION2 = "Section 2 Signature";
//const char Sig3[256] PROGMEM_SECTION3 = "Section 3 Signature";

// DOES verify...
//const char Sig0[256] __attribute__(( __section__(".FLMAP_SECTION0"))) = "Section 0 Signature";
//const char Sig1[256] __attribute__(( __section__(".FLMAP_SECTION1"))) = "Section 1 Signature";
//const char Sig2[256] __attribute__(( __section__(".FLMAP_SECTION2"))) = "Section 2 Signature";
//const char Sig3[256] __attribute__(( __section__(".FLMAP_SECTION3"))) = "Section 3 Signature";

// Does NOT verif...
//const char Sig0[257] PROGMEM_SECTION0 = "Section 0 Signature";
//const char Sig1[257] PROGMEM_SECTION1 = "Section 1 Signature";
//const char Sig2[257] PROGMEM_SECTION2 = "Section 2 Signature";
//const char Sig3[257] PROGMEM_SECTION3 = "Section 3 Signature";

// DOES verify...
//const char Sig0[257] __attribute__(( __section__(".FLMAP_SECTION0"))) = "Section 0 Signature";
//const char Sig1[257] __attribute__(( __section__(".FLMAP_SECTION1"))) = "Section 1 Signature";
//const char Sig2[257] __attribute__(( __section__(".FLMAP_SECTION2"))) = "Section 2 Signature";
//const char Sig3[257] __attribute__(( __section__(".FLMAP_SECTION3"))) = "Section 3 Signature";

// Does NOT verify...
//const char Sig0[512] PROGMEM_SECTION0 = "Section 0 Signature";
//const char Sig1[512] PROGMEM_SECTION1 = "Section 1 Signature";
//const char Sig2[512] PROGMEM_SECTION2 = "Section 2 Signature";
//const char Sig3[512] PROGMEM_SECTION3 = "Section 3 Signature";

// DOES verify...
//const char Sig0[512] __attribute__(( __section__(".FLMAP_SECTION0"))) = "Section 0 Signature";
//const char Sig1[512] __attribute__(( __section__(".FLMAP_SECTION1"))) = "Section 1 Signature";
//const char Sig2[512] __attribute__(( __section__(".FLMAP_SECTION2"))) = "Section 2 Signature";
//const char Sig3[512] __attribute__(( __section__(".FLMAP_SECTION3"))) = "Section 3 Signature";

// Does NOT verify...
//const char Sig0[513] PROGMEM_SECTION0 = "Section 0 Signature";
//const char Sig1[513] PROGMEM_SECTION1 = "Section 1 Signature";
//const char Sig2[513] PROGMEM_SECTION2 = "Section 2 Signature";
//const char Sig3[513] PROGMEM_SECTION3 = "Section 3 Signature";

// Does _NOT_ vereify...
//const char Sig0[513] __attribute__(( __section__(".FLMAP_SECTION0"))) = "Section 0 Signature";
//const char Sig1[513] __attribute__(( __section__(".FLMAP_SECTION1"))) = "Section 1 Signature";
//const char Sig2[513] __attribute__(( __section__(".FLMAP_SECTION2"))) = "Section 2 Signature";
//const char Sig3[513] __attribute__(( __section__(".FLMAP_SECTION3"))) = "Section 3 Signature";

[SpenceKonde]

Like dark matter and the fate of entropy contained in matter falling into a black hole, the nature of this issue is not understood.

There is an order here:

1. Find a motherfucking serial adapter and USB hub that actually fucking work.
2. Use those to figure out what's wrong with serial and fix it on megaTinyCore otherwise the same bug will be ported to DxC.
3. Release next megaTinyCore version.
4. Then begin looking at DxCore.

So far steps 1 and two have stymied me all day. I'm about ready to start chucking USB devices out the door of my room. Does anything in this fucking world work these days??? I'd need to like take a hammer to them first though or they'd make it back into circulation. And I think my roommates would be pissed.

I do not want a USB 3.0 hub. I want a USB 2.0 hub that I don't have to question. One that doesn't need an electrolytic cap sticking out a hole I drilled for it in the case to keep the voltage levels stable, one which doesn't need to be weighed down with a composite of sand and hotmelt glue to keep the force from the wires from holding it up in the air. One that, upon opening it up, won't be found to have only one of the structural pins soldered inplace with the telltake ring of rosin around it indicating that some poor sap in china was being paid to use an iron to solder on the USB sockets...

I do not know if such items exist anymore. There doesn't seem to be such a thing as a quality usb 2.0 hub anymore. And these tests need like 4 serial adapters, my laptop only has 3 USB ports plus a USB C port that's dedicated to the network adapter because some applications of this system involve data transfers to a network drivewhere I really do need the 1 gbit speed. And since the touchpad doesn't work, the touch screen crapped out, a while ago and the third keyboard (which was hell to get because they were no longer made) became unreliable after just a month or two, two of those USB ports are taken up with input devices.

The net result is that I need a way to get from 3 USB ports to around 10 to use this machine comfortably. I have piles of 4-port USB hubs, but the fucking things are unreliable shit and I don''t understand why as technology has advanced, it's become nigh impossible to get a decent USB hub.

Do you have any USB hubs that work that I can buy on amazon for not that much money (I'm unemployed, and spending more time on the cores than I ever did at the stupid day job - that's why I haven't bought a new computer too.

[SpenceKonde]

You know what is way the hell more useful than a list of things that do and don't verify?

A .hex file dumped after the (failed) upload attempt. That actually like, gives me information that would help me to undestanf this.

[pcfreak1201]

Spence, do you have a Amazon Wishlist? Add this https://www.amazon.com/-/en/dp/B001H7AFO6/ref=sr_1_3 and I will make you a gift.

[mosqu-ito]

I'm sorry to hear that things have been giving you Hell lately. I've never known anyone gifted with great intellect who wasn't equally cursed in some other way. Consider the latter to be fate's seal of authenticity on the former. I'm happy to see that you respond to frustration by creating prioritized lists, effective coping strategies are the key to surviving this world. And I do hope you will accept pcFreak's offer! That thing looks to have home-defense grade build quality. I admit that's my own definition, based on whether I think I could use it to cave in someone's skull.

[mosqu-ito]

Knowing how much you love my works/doesn't lists, I've deigned to give you more.

I'm looking at the .map files generated when I compile a flash constant in different ways...

const word Sig[513]    PROGMEM            =   {0x4553, 0x5443, 0x4F49, 0x5F4E, 0x4953, 0x0047};

     // maps at address 000001b4 in .text             Verify successful.

     // Old school, I know how to read it, it works all the way up, and I believe the compiler would tell me if it ran over anything.

const word Sig[513]    PROGMEM_SECTION0   =   {0x4553, 0x5443, 0x4F49, 0x5F4E, 0x4953, 0x0047};

     // maps at address 00804166 in .FLMAP_SECTION0   Verify successful.

     // Apparently Updi knows what to do with that address.  What is it, 24-bit memx?

const word Sig[513]    PROGMEM_SECTION1   =   {0x4553, 0x5443, 0x4F49, 0x5F4E, 0x4953, 0x0047};

     // maps at address 00008000 in .FLMAP_SECTION1   Verify mismatch starting at location 0x008000: 0x53 vs 0xFF

     // This time the address makes sense to me but apparently it confuses Updi.

const word Sig[513]    PROGMEM_SECTION2   =   {0x4553, 0x5443, 0x4F49, 0x5F4E, 0x4953, 0x0047};

     // maps at address 00804166 in .FLMAP_SECTION0   Verify successful.

     // Again, an address Updi seems to like but which confuses me.  Unsurprising, since you seem to alias sections 0 & 2 together

const word Sig[513]    PROGMEM_SECTION3   =   {0x4553, 0x5443, 0x4F49, 0x5F4E, 0x4953, 0x0047};

     // maps at address 00008000 in .FLMAP_SECTION1   Verify mismatch starting at location 0x008000: 0x53 vs 0xFF

     // And you also seem to alias sections 1 & 3 together.  Some arcane knowledge involving bit 15?

const word Sig[513]    __attribute__(( __section__(".FLMAP_SECTION2"))) = {0x4553, 0x5443, 0x4F49, 0x5F4E, 0x4953, 0x0047};

     // maps at address 00010000 in .FLMAP_SECTION2   Verify mismatch starting at location 0x010000: 0x53 vs 0xFF

     // Again, an address that makes sense to me but apparently not to Updi.

const word Sig[513]    __attribute__(( __section__(".FLMAP_SECTION3"))) = {0x4553, 0x5443, 0x4F49, 0x5F4E, 0x4953, 0x0047};

     // maps at address 00018000 in .FLMAP_SECTION3   Verify mismatch starting at location 0x018000: 0x53 vs 0xFF

     // And finally, again here, just the lovely address I would expect, but Updi's unhappy.

[SpenceKonde]

The aliasing is wrong. With it I get wrong output, in the hex file, with tht fixed I get correct output Try changing that part of core_devices to

#if (__AVR_ARCH__ == 104)
  #define PROGMEM_MAPPED   __attribute__(( __section__(".FLMAP_SECTION3")))
  #define PROGMEM_SECTION0 __attribute__(( __section__(".FLMAP_SECTION0")))
  #define PROGMEM_SECTION1 __attribute__(( __section__(".FLMAP_SECTION1")))
  #define PROGMEM_SECTION2 __attribute__(( __section__(".FLMAP_SECTION2")))
  #define PROGMEM_SECTION3 __attribute__(( __section__(".FLMAP_SECTION3")))
#elif (__AVR_ARCH__ == 102)
  #define PROGMEM_MAPPED   __attribute__(( __section__(".FLMAP_SECTION1")))
  #define PROGMEM_SECTION0 __attribute__(( __section__(".FLMAP_SECTION0")))
  #define PROGMEM_SECTION1 __attribute__(( __section__(".FLMAP_SECTION1")))
#else
  // __AVR_ARCH__ == 103, so all of the flash is memory mapped, and the linker
  // will automatically leave const variables in flash.
  #define PROGMEM_MAPPED
#endif

[SpenceKonde]

I know this worked at one time, and I know it wasn't aliasing those sections, so I do not understand how this got fucked up.

[SpenceKonde]

Can you let me know if this solves your problem or if there is still an issue here?

If there is, I think this is an issue that will require collaboration between myself and @qbolsee to resolve.

[mosqu-ito]

The addresses look better but Updi still doesn't like it.

But here's a little good news. Although I'm no closer to understanding the sectioned PROGMEM attributes, it occurred to me that once the flash has been stuffed, be it the old way or the new way, it's then the same until runtime. So I stuffed my flash the old way (keeping both the compiler and Updi happy) and THEN checked so see if I could access it through the window. It works! I did use my own code (code which would no-doubt send you into a fatal fit of non-stop vomitting) because I wanted to be sure I wasn't absorbing "the bug" if you know what I mean.

So I can confirm that once the flash has been stuffed with constants - by whatever means - the actual reading of it through the flash window is very straight forward. The fact that you describe it otherwise makes me wonder if you were resorting to something you didn't really need to resort to.

I would suggest that the first little test you might try would be to ensure that you are indeed successfully changing the flash map (ie. read and display it after every attempt to change it). If a fellow thought he was changing it but wasn't, it might drive him to do some pretty crazy stuff.

[mosqu-ito]

PS. Sorry about the annoying formatting.

PPS. I was a little afraid of insulting your intelligence but I'll risk it and ask if you're familiar with section 11.3.6 of the datasheet. I know a lot of people don't even read as far as section 4.2

[mosqu-ito]

PPPS: Just because I observed Updi doing some multiple upload/verify passes doesn't mean you weren't right in the first place about holes. I'm not sure I've ever tried using the sectioned attributes to make a contiguous image. Maybe I'll look into that tomorrow.

[mosqu-ito]

Holes.

First, instead of .FLMAP_SECTION0/.FLMAP_SECTION1/.FLMAP_SECTION2/.FLMAP_SECTION3 I have to use PROGMEM/.FLMAP_SECTION1/.FLMAP_SECTION2/.FLMAP_SECTION3 That keeps the addresses reasonable.

Then I have to pack everything below .FLMAP_SECTION3 full, no holes.

If I do that, Updi happily uploads and verifies, and I can read it all through the window or old-school. But if I leave the tiniest hole anywhere below .FLMAP_SECTION3 the verify fails just past the hole.

[SpenceKonde]

Yeah, it sounds like serialUPDI can't deal with holes while verifying. That, obviously is bad. Can you try writing with serialUPDI with a hole, and then reading it back out using jtag2updi to dump it to a file - and see if the problem is that SerialUPDI doesn't write after a hole, or that it fails to verify on a hole?

[mosqu-ito]

I don't know how to use jtag2updi. :D But my program still runs even after a verification error and it tells me that there's nothing but a sea of FF beyond the hole, whereas there should be a capital S there. So the verification pass is correct to complain that it found FF where 53 should be. It's the writing that stops at the hole.

[mosqu-ito]

I won't harp on about the CCP unlock sequence, I just thought it might be a valuable thing to bring up if the flash window was causing you grief.

As for the holes, I'm already pretty happy now that I know to just pad them out. I still have to size the three pads manually, which sucks a bit, but otherwise, it works for me.

Perhaps all that is needed for now is a little documentation note to warn people that with the current version of serialUPDI it IS necessary to pad-up the first three sections.

As for the names to use for the sections, maybe there could be a note there too, along the lines of "As an alternative try these section names..."

And if you're feeling uneasy about the windowed access, perhaps just remind people that "legacy flash access" is still supported.

There's nothing so very wrong now that people can't work with it if they know what to do.

[SpenceKonde]

Well this is becoming increasingly pressing, because it blocks everything I want to do with DxCore, which now may actually be working...

[SpenceKonde]

I do not consider padding out 70% of the flash during development to be an acceptable workaround, I guess.

[mosqu-ito]

I know nothing about Python or even how the python modules get called by the IDE. The only little half-baked observation I've made, just reading through some of the pies, is that there is a parameter called "offset" which appears to let you start writing somewhere other than the beginning. Again, I hasten to point out that I don't know what I'm talking about, but do you think there is any possibility of writing four times; once at offset 0, again at 32K, again at 64K, and finally at 96K?

[mosqu-ito]

Okay, I felt bad for not helping more, so I taught myself just enough Python to track this down. The problem is in pymcuprog/serialupdi/nvm.py at line 302. I know you wanted to speed things up there, you young whippersnapper (one of the few good things about being old, I get to say "whippersnapper") but apparently it's not just the first page that needs the check, it's any time the address pointer can't just be left where it was ...like when there are holes.

I changed...

if bulkwrite == 0 or address == 0x800000:

to

if bulkwrite == 0 or (address & 32767) == 0:

Now you can have all the holes you like. It writes (all of it) verifies, and runs.

[mosqu-ito]

...of course if you find a way to make a hole mid-section you'll need a more elaborate conditional.

[SpenceKonde]

Right but this would catch the main cause..

I'm trying to test it now, but suddenly serial has completely ceased all functioning and I didn';t touch any file involved with serial so I have no idea whaty the fuck is goign ion

[SpenceKonde]

Tested. Works. Serial issue appears to have not been real in the first place.