Flash too limited to allow OLED & SDCard

[mmalluck]

It's possible to free up and additional 1.5k Flash by loading the latest Uno Bootloader onto the Nano. This may provide enough space to allow both the OLED and SDCard to operate together.

Reference: https://arduino.stackexchange.com/questions/51866/arduino-nano-atmega328p-bootloader-difference

With an update to the Nano bootloader, the good folks at Arduino neglected to reduce bootloader partition from 2Kb to 500b, as was done for the Uno. Loading the Uno bootloader onto the Nano provides this benefit. You just have to remember to identify the board as an Uno in the IDE.

If you have a sketch with both the OLED and SDCard functions active, you could see if it properly compiles for the Uno. If so, it would work for the modified Nano.

[spenceraxani]

Hi,

That’s really neat, I’ll have to look into it.

My guess is that it frees up the space on the EEPROM, which won’t be particularly useful to us. We probably require about 10 kB of extra flash storage and about 1 kB of SRAM in order to run both the OLED screen and the SD card at the same time.

We actually only use the EEPROM to store the name of the detector.

Thanks,

Spencer

On Jun 27, 2018, at 11:52 AM, mmalluck notifications@github.com wrote:

u just have to

[mmalluck]

Hello Spencer,

The 1.5k is indeed flash (not eeprom) space. I found another reference that provides a how-to on uploading the Uno bootloader to the Nano, just to take advantage of the extra program space.

http://www.buildlog.net/blog/2017/09/getting-more-program-space-on-an-arduino-nano/

As far as what's needed to run a combined OLED & SD implementation, a quick mashup of the two sketches weighs in around 35k.... which is 3k more than the modified nano or Uno could give us. And the ram is 116bytes over. Drat.

For fun I may try to see if this can be optimized.

Thanks,

Michael Malluck

On Wed, Jun 27, 2018, 2:21 PM Spencer N. Axani notifications@github.com wrote:

Hi,

That’s really neat, I’ll have to look into it.

My guess is that it frees up the space on the EEPROM, which won’t be particularly useful to us. We probably require about 10 kB of extra flash storage and about 1 kB of SRAM in order to run both the OLED screen and the SD card at the same time.

We actually only use the EEPROM to store the name of the detector.

Thanks,

Spencer

On Jun 27, 2018, at 11:52 AM, mmalluck notifications@github.com wrote:

u just have to

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spenceraxani/CosmicWatch-Desktop-Muon-Detector-v2/issues/10#issuecomment-400781924, or mute the thread https://github.com/notifications/unsubscribe-auth/Amhe4thPymCkiD1hZ5y8Jgl9LW-sD97Oks5uA81AgaJpZM4U6DFP .

[spenceraxani]

There’s strange behavior with the Arduinos that don’t actually allow you to go above ~70% of either the maximum of the SDRAM or the Flash.

I bet there’s ways to optimize either the OLED library or the SD Library to make them work. I’ve tried my best to optimize the Arduino code, but I’m sure there’s always places for improvement.

On Jun 27, 2018, at 2:22 PM, mmalluck notifications@github.com wrote:

Hello Spencer,

The 1.5k is indeed flash (not eeprom) space. I found another reference that provides a how-to on uploading the Uno bootloader to the Nano, just to take advantage of the extra program space.

http://www.buildlog.net/blog/2017/09/getting-more-program-space-on-an-arduino-nano/

As far as what's needed to run a combined OLED & SD implementation, a quick mashup of the two sketches weighs in around 35k.... which is 3k more than the modified nano or Uno could give us. And the ram is 116bytes over. Drat.

For fun I may try to see if this can be optimized.

Thanks,

Michael Malluck

On Wed, Jun 27, 2018, 2:21 PM Spencer N. Axani notifications@github.com wrote:

Hi,

That’s really neat, I’ll have to look into it.

My guess is that it frees up the space on the EEPROM, which won’t be particularly useful to us. We probably require about 10 kB of extra flash storage and about 1 kB of SRAM in order to run both the OLED screen and the SD card at the same time.

We actually only use the EEPROM to store the name of the detector.

Thanks,

Spencer

On Jun 27, 2018, at 11:52 AM, mmalluck notifications@github.com wrote:

u just have to

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spenceraxani/CosmicWatch-Desktop-Muon-Detector-v2/issues/10#issuecomment-400781924, or mute the thread https://github.com/notifications/unsubscribe-auth/Amhe4thPymCkiD1hZ5y8Jgl9LW-sD97Oks5uA81AgaJpZM4U6DFP .

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/spenceraxani/CosmicWatch-Desktop-Muon-Detector-v2/issues/10#issuecomment-400799746, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpeu5trm3dZVvk9HifYrPByM5_zH6nQks5uA9uNgaJpZM4U6DFP.

[jgOhYeah]

I currently trying to get the program to work with a smaller library for the oled displays. The library can be found here: https://github.com/greiman/SSD1306Ascii The compiled sketch with the sd library and this library uses around 57% of the global RAM and 93% of the program memory. This is with the default arduino nano bootloader. As for RAM, I have heard that using char arrays instead of the String library makes ram usage more predictable and stable as ram is not dynamically allocated. My attempts can be found here: https://github.com/jgOhYeah/NonOfficial-StuffForCosmicWatchMuonDetectors

[spenceraxani]

That’s really neat! It looks like you are getting close. Keep at it and let me know if you come up with a solution. 23% less program space seems almost achievable.

On Jun 28, 2018, at 7:15 PM, jgOhYeah notifications@github.com wrote:

I currently trying to get the program to work with a smaller library for the oled displays. The library can be found here: https://github.com/greiman/SSD1306Ascii https://github.com/greiman/SSD1306Ascii The compiled sketch with the sd library and this library uses around 57% of the global RAM and 93% of the program memory. This is with the default arduino nano bootloader. As for RAM, I have heard that using char arrays instead of the String library makes ram usage more predictable and stable as ram is not dynamically allocated. My attempts can be found here: https://github.com/jgOhYeah/NonOfficial-StuffForCosmicWatchMuonDetectors https://github.com/jgOhYeah/NonOfficial-StuffForCosmicWatchMuonDetectors — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/spenceraxani/CosmicWatch-Desktop-Muon-Detector-v2/issues/10#issuecomment-401210324, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpeuwsK0j1jnm8pq1uAO5edx75EyN_tks5uBXGYgaJpZM4U6DFP.

[mmalluck]

Hello,

That's funny. I was playing with the SSD1306 library this afternoon too. While it does save a bunch of ram and considerable flash, the execution time is pretty poor.

I just have the V1 hardware at the moment so working with older sketch, baselining the 'downtime' with adafruit_SSD1306 library comes in around 3% of the total execution cycle. The SSD1306ascii comes around 12%.

So the ascii library takes 4 times longer to execute than the adafruit. I guess there's no such thing as a free lunch; You can be space efficent or execution efficient. Is this acceptable? I guess that depends on what you want.

I might start looking at the u8g2 libraries next.

Regards, Michael

On Thu, Jun 28, 2018, 8:15 PM jgOhYeah notifications@github.com wrote:

I currently trying to get the program to work with a smaller library for the oled displays. The library can be found here: https://github.com/greiman/SSD1306Ascii The compiled sketch with the sd library and this library uses around 57% of the global RAM and 93% of the program memory. This is with the default arduino nano bootloader. As for RAM, I have heard that using char arrays instead of the String library makes ram usage more predictable and stable as ram is not dynamically allocated. My attempts can be found here: https://github.com/jgOhYeah/NonOfficial-StuffForCosmicWatchMuonDetectors

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/spenceraxani/CosmicWatch-Desktop-Muon-Detector-v2/issues/10#issuecomment-401210324, or mute the thread https://github.com/notifications/unsubscribe-auth/Amhe4vBRBIWtSiftNV7KAaQdb15FYBuaks5uBXGYgaJpZM4U6DFP .

[jgOhYeah]

I just checked one of the files logged to the sd card and the dead time is around 43% of the total time after 25 hours running. Might have to speed it up slightly somehow. I am investigating other libraries which may be a compromise between the two. I have measured the standard oled version of the software to have a dead time of around 4% and the sd card only version with an sd card inserted to be around 0.3% This might be one of the reasons why the detector I built does not always seem to be behaving properly :)

[mmalluck]

Considering you can pack all the code into a single sketch, you could take some unused I/O pins and a switch and let the operator decide what he wants to run in software; Serial / OLED & Serial / SD & Serial / OLED & SD & Serial. This way you don't need to upload three different sets of code and the user can determine what kind of performance vs features he wants.

[jgOhYeah]

I have swapped to the u8x8 library for driving the display, which is a part of the u8g2 library and is text only. It seems to be fairly fast and not use that much memory. For those interested I have attached a spreadsheet of tests I have done comparing a few libraries. The results seem to match up with Michael's measurements for dead time differences between the libraries. SSD1306Libraries.xlsx Dead time seams to be around 5% of the total time now with both the screen on, logging to serial and the sd card. Here it is: https://github.com/jgOhYeah/NonOfficial-StuffForCosmicWatchMuonDetectors

By the way, the reason why I measured 40% dead time for the SSD1306ascii library was probably because I put a bracket in the wrong place and it was probably trying to do adc readings on the wrong pin. :)

[spenceraxani]

Great stuff. I’ve tried that library before and couldn’t get it working. We’ll upload your code today to test it out. Great comparison in the excel spreadsheet by the way.

Thanks,

Spencer

On Jul 7, 2018, at 5:58 AM, jgOhYeah notifications@github.com wrote:

text

[mmalluck]

I just wanted to provide an update. I managed to get both OLED and SDcard functions working within the same software set. I had to swap out the adafruit library for the SSD1306Ascii library, make a few modifications to the display calls and cludge-in the existing SD card functions. My code is not pretty, but works as a proof of concept.

It uses 97% of the program memory and 65% of the dynamic memory. Not much more could be squeezed out. Feel free to use my proof of concept code however you would like. I'm just happy helping out the project.

Cosmic_Watch_V2_Comb_Simp_Disp.zip

[spenceraxani]

Hi,

Haha, thanks for working on squeezing both pieces of code onto the Arduino. You are really hitting the limits of the Arduino Nano with your code. The dynamic memory was often an issue for us, since issues can arise when this increases beyond about 70%.

Glad to hear you got it working though. I’ll try to test it out soon and see if I have any useful comments.

Thanks,

Spencer

On Apr 25, 2019, at 7:53 AM, mmalluck notifications@github.com wrote:

I just wanted to provide an update. I managed to get both OLED and SDcard functions working within the same software set. I had to swap out the adafruit library for the SSD1306Ascii library, make a few modifications to the display calls and cludge-in the existing SD card functions. My code is not pretty, but works as a proof of concept.

It uses 97% of the program memory and 65% of the dynamic memory. Not much more could be squeezed out. Feel free to use my proof of concept code however you would like. I'm just happy helping out the project.

Cosmic_Watch_V2_Comb_Simp_Disp.zip https://github.com/spenceraxani/CosmicWatch-Desktop-Muon-Detector-v2/files/3116911/Cosmic_Watch_V2_Comb_Simp_Disp.zip — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/spenceraxani/CosmicWatch-Desktop-Muon-Detector-v2/issues/10#issuecomment-486659989, or mute the thread https://github.com/notifications/unsubscribe-auth/AANF5O3GDXA66KQN7624FPLPSGSWHANCNFSM4FHIGFHQ.

[mmalluck]

Sounds good. I should mention upon the first upload you may have some garbage displayed on the OLED that will persist until all power is removed. It has something to do with the OLED not liking seeing the libraries swapped out.

[jgOhYeah]

Hello, Good job on modifying the code and getting it to work. I eventually gave up on the library for this job myself because I thought it was just too slow, but obviously not :).

I also wrote a version that has been running on my physics teacher's detectors for a few months, but I forgot to upload the latest version until now. Sorry for the slight delay. It should be here https://github.com/jgOhYeah/NonOfficial-StuffForCosmicWatchMuonDetectors.

I used the u8x8 library, which is much more limited in terms of fonts and font sizes, but is fast and does not use much memory, as well as being versatile enough to make something readable. If it is possibly to use a slightly nicer looking, low memory, high speed library, that is good to know :).

Last time I checked, the dead time for this code with display and sd card was about 8%. I am not sure if this good enough, but it seems to work. This code uses 91% of the program memory and 65% of the ram, so it seems that no matter the libraries or code used, it is always going to be tight squeezing in and sd card and display.

There was somehow enough room left that I managed to put in a module that can burn the detector's name and a few other settings into the EEPROM without having to upload a different piece of code to it.

I have read that with the ram in arduinos, it is often better to use char arrays instead of the String class as the String class can waste memory, eventually running out and causing the arduino to crash. I found an interesting article on it here. Maybe with stuff like this in mind it would be possible to successfully go above the 70% dynamic memory suggested limit if needed?

[mmalluck]

Hello jgOhYeah,

Your code is so pretty, polished, and well commented. Mine was just a kludge to see if the concept would work. I'm pretty sure I broke all of the external tools as my print statements / orders differ from what the external tool expects, but nothing that couldn't be fixed.

SSD1306Ascii library is pretty good as it's written to be very low memory and they have a number of fonts to choose from. The arial14 font I chose is pretty nice looking and has the same character sizing as the adafruit library.

With regards to string and ram, I saw a similar article that discussed how the arduino compiler will allocate ram for any strings because it operated under the notion that they may be dynamically manipulated elsewhere in the code. This can be very wasteful if your using a bunch of static strings, like we do here. I like how you used char arrays to keep the ram usage down, but there's actually a built in function in the arduino IDE that can help with a little less effort. It's a function call F(). By defining your string inside this wrapper function, the string will get statically allocated in program memory and not ram. That's how I kept my ram usage down. You can even use it with normal string operations for example Serial.println( F("Your string here") + F("Your other string") + somenumber );

It would be interesting to see if how the char array method and the F() compare against each other with regards to ram and cpu usage.

[jgOhYeah]

Semi-organised code and extra features are mainly a result of trying to avoid homework :) Although it often starts out with comments and nicely layed out, it often ends up very messy with very out of date comments making little sense :) The notes are also because I have been known to leave projects for a few years or so and completely forget what I was doing when I came back to them and have to start from scratch.

I have not done any tests speed wise, but I would have it a guess that the F() macro is doing something similar to the strcpy_P in the background in that it copies strings saved into flash into char arrays, but does it automatically. Probably the fastest is the Serial.println("text") without anything to save memory as the string is already in ram.

In another project I have been using the F() macro and it makes it much easier to follow, however in this code I wanted to write identical strings to up to 3 separate things (serial, display and sd card) using the least amount of memory possible.

I ran a small test comparing memory usage of different ways of saving and then printing a string out twice and found that the F() macro saves the string into flash memory twice, even if both strings are identical. This surprises me, because I would have thought the compiler would optimise this, but evidently not.

By using strcpy_P, the string can be saved into flash and then only copied into a char array only when required, allowing a string to be used multiple times if required, although it is very manual to set up. It does also allow char arrays to be manually manipulated and reused, as well as a scratchpad for processing hopefully without having to allocate more memory.

Once I started with this system, I decided to keep going with it as to have all UI text on the one file, even if strings are only used once for consistency. Not sure if this is the best way to do it, but that's what I did.

StringMemTest.zip.

As for the compatibility with the existing python scripts programs, that only happened much later in the version I did and as of now I still have not got round to implementing the reading and deleting files from the sd card bit. A work in progress :)

[jgOhYeah]

A slight update to my version of the code:

I only recently noticed that in February 2019 there was an update (v2.25.10) to the U8x8 library that claims it should be around 4% faster (see near the bottom here). I have not tested this in this application, however I was playing around with drawing images on the screens and ended up with a graphical bar graph and scale instead of the hyphen one. It ranges from 0 to 180mV and each division should be around 20mV (off by a bit due to rounding to fit inside 120px wide).

Interestingly, early testing suggested that loading the top image background from flash into ram, overlaying the bar on top of it at the right length and then displaying it somehow takes around 1/3 the time it does for my adaptation of the hyphenated version. Not really sure why as I thought that the hyphenated version would have been more efficient, might be something to do with how the library handles characters in the background?

On the other hand all of these do not seem to have major effects (fractions of a % only) with dead times of:

• 4.6% old (v2.2) for 24 hours in slave mode with a rate of 0.121c/s (not 100% sure which version of U8x8, probably pre v2.25.10)
• 4.1% new (v2.3.2) for around 17.5 hours in slave mode with a rate of 0.138c/s (U8x8 v2.26.14)

I have updated the non-official repository with the latest version. Good luck with your muon detecting :)

[spenceraxani]

Hi,

Thanks for the update about the library.

Spencer

On Aug 13, 2019, at 7:47 AM, jgOhYeah notifications@github.com wrote:

A slight update to my version of the code:

I only recently noticed that in February 2019 there was an update (v2.25.10) to the U8x8 library that claims it should be around 4% faster (see near the bottom here https://github.com/olikraus/u8g2/blob/2ce3043cf43ab1c47de85f5a06b55eaf93700aff/ChangeLog). I have not tested this in this application, however I was playing around with drawing images on the screens and ended up with a graphical bar graph and scale instead of the hyphen one. It ranges from 0 to 180mV and each division should be around 20mV (off by a bit due to rounding to fit inside 120px wide).

Interestingly, early testing suggested that loading the top image background from flash into ram, overlaying the bar on top of it at the right length and then displaying it somehow takes around 1/3 the time it does for my adaptation of the hyphenated version. Not really sure why as I thought that the hyphenated version would have been more efficient, might be something to do with how the library handles characters in the background?

On the other hand all of these do not seem to have major effects (fractions of a % only) with dead times of:

4.6% old (v2.2) for 24 hours in slave mode with a rate of 0.121c/s (not 100% sure which version of U8x8, probably pre v2.25.10) 4.1% new (v2.3.2) for around 17.5 hours in slave mode with a rate of 0.138c/s (U8x8 v2.26.14) I have updated the non-official repository with the latest version. Good luck with your muon detecting :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/spenceraxani/CosmicWatch-Desktop-Muon-Detector-v2/issues/10?email_source=notifications&email_token=AANF5O6HYRRWLVWDHO476U3QEKNMHA5CNFSM4FHIGFH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4FM6QA#issuecomment-520802112, or mute the thread https://github.com/notifications/unsubscribe-auth/AANF5OZMFHVHBB5ZPHLQF2LQEKNMHANCNFSM4FHIGFHQ.