Request for SPIFFS support for ESP8266

[judge2005]

Hi,

Would you consider adding support for the creation and upload of SPIFF bin files?

Thanks - Paul

[jantje]

I think this has been asked before but I can't recall where or by who. I would consider it but it is not high on my agenda (actually at this time it is not)

[judge2005]

That's OK. Could you give me hints as to how I might add this myself? I am a professional Java developer and long time user of Eclipse, though I have never worked on plugins for Eclipse.

[jantje]

The best thing is look for java code that does it. Which in this case probably exists. Then integrate this code into the plugin. Then make a pull request so that later versions still contain the code

[jantje]

I can help with the integration where when how what

[wimjongman]

Hi Paul. Can You sketch out how you want to approach this? For example, what menu items go where, any mockup of screens, what is the output, etc..

On Jan 14, 2017 14:15, "Paul Andrews" notifications@github.com wrote:

That's OK. Could you give me hints as to how I might add this myself? I am a professional Java developer and long time user of Eclipse, though I have never worked on plugins for Eclipse.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jantje/arduino-eclipse-plugin/issues/628#issuecomment-272623494, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC6ZX5gE41BCprNXTBSfQ0FCVxU8v9ks5rSMpygaJpZM4LjfMd .

[judge2005]

Hi,

I forked and cloned master, I have one compilation error in BoardSelectionPage line 306:

Description Resource Path Location Type The method requestLayout() is undefined for the type Composite BoardSelectionPage.java /io.sloeber.ui/src/io/sloeber/ui/project/properties line 306 Java Problem

@wimjongman I am a long way from figuring that out. If I can get anything to work at all in a reasonable timeframe I will get back to you.

[judge2005]

I figure I will make this work like the equivalent process in the Arduino IDE. i.e. everything in a directory called 'data' will be packaged up by mkspiffs and then uploaded using esptool (or OTA).

I see that mkspiffs is defined in platform.txt, somewhat like the esptool and network upload tool is. I see that 'esptool' is defined in boards.txt as 'generic.upload.tool', but I don't see how the path to the executable is resolved at runtime. I see that network_cmd is referenced in pre_processing_platform_default.txt, but I don't see the key (tools.esp8266OTA) being referenced anywhere in the code. In short, I'm not sure how I am supposed to find the path to mkspiffs. When I figure this out (or you tell me) I will then also have to figure out:

1. where to put the code that invokes it.
2. What causes it to get invoked (say from the arduino menu).
3. Where the resulting package should go.
4. How to invoke esptool (or OTA) to upload it (by which I mean, what code do I invoke, rather than what arguments to provide).

So, any suggestions you have on any of this would be gratefully received.

[jantje]

Description Resource Path Location Type The method requestLayout() is undefined for the type Composite BoardSelectionPage.java /io.sloeber.ui/src/io/sloeber/ui/project/properties line 306 Java Problem

requestLayout is from the class org.eclipse.swt.widgets control. In my code composite extends Scrollable Scrollable extends Control

requestLayout is not so important. It makes the page to recalculate and adapt so it shows ok. resizing the page does the same thing. But it is suspicious that you get this error. It looks like something is wrong which may cause other havoc.

but I don't see the key (tools.esp8266OTA) being referenced anywhere in the code.

That is absolutely normal. The only tool Sloeber knows (from the top of my head) is make and upload. Sloeber works on what it finds in platform.txt files and boards.txt files. It knows the RECIPE(.OBJCOPY) keyword. Sloeber converts the platform.txt and boards.txt into environment variables. you can see the output of this process in project properties->C/C++ build->Environment.

In short: to start mkspiffs the right command is ${A.TOOLS.MKSPIFSS.PATH}/${A.TOOLS.MKSPIFFS.CMD} In code this would be

final private String  MKSPIFFS_PATH="mkspiffs.path";
final private String  MKSPIFFS_CMD="mkspiffs.cmd";
....
String runMkSpiffsCmd=makeEnvironmentVar(MKSPIFFS_PATH)+"/"+makeEnvironmentVar(MKSPIFFS_CMD)
;

To run this use the consoled commands aConsole.RunConsoledCommand(this.myConsole, runMkSpiffsCmd, monitor);

Now some questions from me to you. Ths mkspiffs command is this part of the upload? If so before or after the upload or don't care? Is this a parameterless command? Can't be as it needs the port and the folder and probably more. Do not hard code this info but put in in the prePlatform.txt file. I would go for something like recipe.uploadFiles="{mkspiffs.path}/{mkspiffs.cmd}" -p "{serial.port}" "{build.path}/../data"

PS We use slack now. I have send you an invite.

[judge2005]

Hi,

I will dig some more into the compilation error - I have commented it out for now. FYI I am using Mars, maybe that has some impact?

The info you provided is useful, I should be able to make some progress from that. BTW mkspiffs is a tool that builds the spiffs from one or more files on your computer. esptool is then used to upload the results, so mkspiffs is like the compilation/packaging phase and esptool is the upload phase. Both tools are part of the esp8266 boards package that can be installed in the Arduino IDE.

Both tools will require a lot of command line options. I am basing a lot of this on https://github.com/esp8266/arduino-esp8266fs-plugin.

[jantje]

Mars could be the reason. The comment states

• @since 3.105 Looks like a really long time ago as far as I see

To make the mkspiff part of the build add following line to the pre_platform.txt recipe.objcopy.mkspiffs="{mkspiffs.path}/{mkspiffs.cmd}" xxxx "{build.path}/../data" yyyy This makes it part of the post build steps. This will probably cause some havoc on other boards but let me think/worry for that right now.

From the link you supplied I would conclude they run the upload spiff as a command not connected to the build or the upload. Given that I would use the external tools but it is not easy to get the cdt environment variables there. However it is easy to reference a "installed tool" as the command line is not encoded in the platform.txt this could be a quick and easy way

[judge2005]

OK. I am thoroughly confused. The nearest filename to pre_platform.txt is pre_processing_platform_default.txt and there is nothing like the entry you give in there. I really want to do this, but I am thinking at the moment that the easiest thing to do would be to configure a couple of external tools. The paths can be made relative to eclipse_home. e.g. ${eclipse_home}/arduinoPlugin/tools/esp8266/mkspiffs/0.1.2/mkspiffs.exe with arguments specific to my situation.

[wimjongman]

Never settle Paul! ;)

On Sun, Jan 15, 2017 at 10:50 PM, Paul Andrews notifications@github.com wrote:

OK. I am thoroughly confused. The nearest filename to pre_platform.txt is pre_processing_platform_default.txt and there is nothing like the entry you give in there. I really want to do this, but I am thinking at the moment that the easiest thing to do would be to configure a couple of external tools. The paths can be made relative to eclipse_home. e.g. ${eclipse_home}/arduinoPlugin/tools/esp8266/mkspiffs/0.1.2/mkspiffs.exe with arguments specific to my situation.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jantje/arduino-eclipse-plugin/issues/628#issuecomment-272726579, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC6UP4Qfy9SvaUj93om-c7449OFt8Vks5rSpS3gaJpZM4LjfMd .

[jantje]

OK. I am thoroughly confused. The nearest filename to pre_platform.txt is pre_processing_platform_default.txt and there is nothing like the entry you give in there.

I said :"add following line to the pre_platform.txt" So yes indeed it is not there.

As to the name any existing pre_platform file will do. I'm not sure whether it is already implemented but pre_processing_platform_8266.txt would be the best choice. But given the current situation I would go with pre_processing_platform_default.txt.

Platform.txt and boards.txt It is very simple and yet also so complex. A long time ago I wrote 2 posts to explain the basics. They are still recommended to level up. http://blog.baeyens.it/#post22 http://blog.baeyens.it/#post24 I added the default pre and post boards and platform .txt file. I think the names are pretty self explaining when you levelled up.

external tools Yesterday I looked into the external tools and the problem I see there is that there is no project known in external tools. So you would have to create a external tool for each project you want to uses mkspiffs. You would also have to keep the settings in sync. Which is a solution but only for people who really know what they are doing @wimjongman I really think a currently active project would be a great addition to eclipse. I wouldn't care it to be part of Sloeber but it is a commonly sought for feature that really makes sense. Just like "active configuration" in cdt and the mylin task list has active tasks as well. Once we have a active project we could have a external cdt task adding the active project active configuration build environment vars to the environment vars to launch kind of like any recipe from the platform.txt files.

[judge2005]

Just to let you know; I haven't forgotten, but life has taken over! Very busy with my son's robotics club. It also turns out that it was good to not jump straight in, because I now have a list of esp8266 related commands I want to add:

1. mkspiffs - to generate the spiffs.
2. upload spiffs - using the regular (wired) tool.
3. upload spiffs OTA
4. erase flash

All of which I have implemented as external commands, but that is very clunky!

[atanisoft]

judge2005, for your issue #1 (mkspiffs) and #2 (upload, wired) you can look at https://github.com/esp8266/arduino-esp8266fs-plugin/blob/master/src/ESP8266FS.java which is a plugin for the Arduino IDE that handles these.

[judge2005]

I added the following lines to ... in my local copy. This is really just to document what the 'recipe' is as I doubt that this is how it should really be done - i.e. it isn't really part of the build. It seems like these should be on some sort of context menu so that the user can select the resources that would be the target of the command. They should also only be enabled for ESP8266 projects and the spiff uploading should switch between OTA and serial based on what port is selected for the uploading. Perhaps it would be easier if the existing environment variables were accessible to external command scripts using the env_var eclipse variable (or are they already - I just tried it, they don't seem to be)?

for erase flash. Only works for ESP8266 and only if using wired...

A.RECIPE.OBJCOPY.ERASE_FLASH={A.TOOLS.ESPTOOL.PATH}/{A.TOOLS.ESPTOOL.CMD} -cd ck -cb {A.UPLOAD.SPEED} -cp {A.SERIAL.PORT} -ce

for mkspiffs. Should include -s {{A.BUILD.SPIFFS_END} - {A.BUILD.SPIFFS_START}}. Or perhaps this should be a parameter? As should the source and destination.

A.RECIPE.OBJCOPY.MKSPIFFS={A.TOOLS.MKSPIFFS.PATH}/{A.TOOLS.MKSPIFFS.CMD} -c {A.BUILD.PATH}/../data -p {A.BUILD.SPIFFS_PAGESIZE} -b {A.BUILD.SPIFFS_BLOCKSIZE} spiffs

for wired spiff upload. Should parameterize spiff file name

A.RECIPE.OBJCOPY.UPLOAD_SPIFF={A.TOOLS.ESPTOOL.PATH}/{A.TOOLS.ESPTOOL.CMD} -cd ck -cb {A.UPLOAD.SPEED} -cp {A.SERIAL.PORT} -ca {A.BUILD.SPIFFS_START} -cf {A.BUILD.PATH}/../spiffs

for OTA spiff upload. Only works for ESP8266 and only if using OTA. Should parameterize spiff filename

A.RECIPE.OBJCOPY.UPLOAD_SPIFF_OTA={A.TOOLS.ESPTOOL.NETWORK_CMD} {A.RUNTIME.PLATFORM.PATH}/tools/espota.py -i {A.SERIAL.PORT} -p {A.NETWORK.PORT} --auth={A.TOOLS.ESPTOOL.NETWORK.PASSWORD} -s -f spiffs

[jantje]

If these are the commands to be run... wouldn't it be a good idea to add them as programmers?

[mpandrews2000]

I think mkspiffs is part of the build. Uploading the spiffs isn’t, but it would be painful to have to switch to a different programmer when you wanted to upload them and then back again when you wanted to upload a sketch. Clearing flash is essentially resetting the ESP to factory defaults.

From: jantjemailto:notifications@github.com Sent: Tuesday, March 7, 2017 8:31 PM To: Sloeber/arduino-eclipse-pluginmailto:arduino-eclipse-plugin@noreply.github.com Cc: Paul Andrewsmailto:mpandrews@outlook.com; Commentmailto:comment@noreply.github.com Subject: Re: [Sloeber/arduino-eclipse-plugin] Request for SPIFFS support for ESP8266 (#628)

If these are the commands to be run... wouldn't it be a good idea to add them as programmers?

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/Sloeber/arduino-eclipse-plugin/issues/628#issuecomment-284917655, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AIsNqx26P2NGnCM-OecQRpzzRoNZzXu1ks5rjgT8gaJpZM4LjfMd.

[atanisoft]

@judge2005, In your A.RECIPE.OBJCOPY.ERASE_FLASH and A.RECIPE.OBJCOPY.UPLOAD_SPIFF variables it would be better to use A.UPLOAD.RESETMETHOD instead of hardcoding ck. This is configurable by the user in project properties. Instead of hardcoding this: {A.BUILD.PATH}/../data you can instead use ${ProjDirPath}/data. Using spiffs (no extension) is a bit cryptic and would be better as spiffs.bin.

[jantje]

@mpandrews2000 If I understand this correctly it is more like the "burn bootloader". Maybe we could have a file with commands called commands.txt added to the core?

@judge2005

Perhaps it would be easier if the existing environment variables were accessible to external command scripts using the env_var eclipse variable (or are they already - I just tried it, they don't seem to be)?

I'm with you here but it is not so easy. env_var is a eclipse concept. So even-though all these environment Variables are in CDT->configuration, configuration is a CDT concept (not core eclipse) and as such not reachable. But it is worse: I have added env_var variables that turned out not be be available at other locations in eclipse where you can use env_vars.

[bebo-dot-dev]

here's what I did (on linux) and it's not exactly rocket science but in the absence of anything else, it works well enough for me. In Neon -> Run -> External Tools -> External Tools Configurations:

1. Create an "ESP8266 mkspiffs" external tool configuration:
   • Name: ESP8266 mkspiffs
   • Location: /path/to/mkspiffs
   • Working Directory: ${workspace_loc:/your_project_folder}
   • Arguments: -c ${workspace_loc}/your_project_folder/data -p 256 -b 8192 -s 3125248 ${workspace_loc}/your_project_folder/your_project.spiffs.bin
2. Create an "ESP8266 spiffs upload" external tool configuration:
   • Name: ESP8266 spiffs upload
   • Location: /path/to/esptool
   • Working Directory: ${workspace_loc:/your_project_folder}
   • Arguments: -cd none -cb 115200 -cp /dev/ttyUSB0 -vv -ca 0x100000 -cf ${workspace_loc}/your_project_folder/your_project.spiffs.bin

You can even setup a "wipe flash" tool too by calling esptool with the correct parameters.

I realize that this is a little rigid and there may be a way to better determine the argument parameters in a more dynamic fashion than I've implemented here.

[jantje]

I think I need to dive into these external tools. It may contain an easy fix.

[bebo-dot-dev]

Since erase flash was mentioned in https://github.com/Sloeber/arduino-eclipse-plugin/issues/628#issuecomment-277052194, here are the arguments to make that work:

3. Create a "Wipe Flash" external tool configuration:
   • Name: Wipe Flash
   • Location: /path/to/esptool
   • Working Directory: ${workspace_loc:/your_project_folder}
   • Arguments: -cd none -cb 115200 -cp /dev/ttyUSB0 -vvv -ce

You end up with something like this in the menu when you configure these:

For me so far as I can tell, the configuration for these "configuration based jobs" get stored in xml files in /home/joe/Dev/Workspaces/Eclipse_Neon/.metadata/.plugins/org.eclipse.debug.core/.launches where there's one file present in here per configured job i.e.

• ESP8266 mkspiffs.launch
• ESP8266 spiffs upload.launch
• Wipe Flash.launch etc..

..so with that in mind there might be a way to programatically control these items.

One thing I do see however is that my external tools are always present in the menu regardless of the project type being worked on in eclipse..so that might not be ideal / might have to be worked around.

[bebo-dot-dev]

oh and to eliminate any confusion/ambiguity I guess I should say that where I've mentioned "esptool", I'm referring to this one: https://github.com/igrr/esptool-ck and not some other random esptool (although admittedly it does all appear to be a bit random / wild west in the esptool area ;))

[jantje]

I fear the external tools will not be so easy (for me)

[bebo-dot-dev]

Ah well no probs. As I mentioned earlier in the absence of anything else, this manual lashed together solution works well enough for me right now.

Other people might want a more "professional" implementation and there might be a better way to do it than to leverage external tools..

[ghislainborremans]

Hello has this gone somewhere? it is indeed a nice feature to be able to upload files. straight from Sloeber. Now it is a bumpy way around.

[jantje]

I did not make more progress. Did you try the external tools?

[judge2005]

I have created an ant file that will do these four tasks. It is available at https://github.com/judge2005/sloeber_spiffs. It uses a small inline groovy script to parse the org.eclipse.cdt.core.prefs file, so it will use whatever settings you have defined for the project it runs in, and it picks up the locations of all of the needed external tools and your project from there.

There are a couple of properties in the build.xml that define the spiff source directory and the destination file name.

BTW I couldn't get OTA uploads to work in the brief amount of testing I did. YMMV. and I haven't tested the erase flash task. I didn't want to erase my flash...

Maybe this could be implemented within the plugin itself. On the other hand, it works pretty well as an ant build.

[ghislainborremans]

I wanted to use the ant file here above, but when building i get an null pointer exception in build.xml 20 I do not know ant so i cannot be of big help. I see echo $eclipse_home just before. I have no eclipse_home. i think with sloeber, this is not used??

[judge2005]

I updated the readme with more instructions.