search

nodemcu / nodemcu-firmware

Lua based interactive firmware for ESP8266, ESP8285 and ESP32
https://nodemcu.readthedocs.io
MIT License
7.64k stars 3.12k forks source link

LFS Quick Start Documentation for Docker Builds #2457

Closed jmd13391 closed 6 years ago

jmd13391 commented 6 years ago

8<------------------------ DOCUMENTATION REQUEST ------------------------------------

Missing Documentation

LFS Quick Start (Step by Step) Documentation for application developers using the Docker NodeMCU platform (https://hub.docker.com/r/marcelstoer/nodemcu-build/). This request is related to #2431 (Improving LFS Documentation).

Justification

Currently, the setup and use of LFS is not intuitive to newbie application developers wanting to leverage the ESP8266 NodeMCU platform. Recently, LFS Quick Start documentation (#2432) for application developers using the Cloud Build Service has been introduced in the NodeMCU Dev branch (https://github.com/nodemcu/nodemcu-firmware/blob/dev/docs/en/lfs.md). While this is an excellent start, it does not translate smoothly over to application developers using the Docker NodeMCU Build approach (specifically, the section titled "Build the LFS image").

Workarounds

n/a

8<------------------------ END DOCUMENTATION REQUEST --------------------------------

jmd13391 commented 6 years ago

RE: #2443 - specifically: https://github.com/nodemcu/nodemcu-firmware/issues/2443#issuecomment-409189592

Running that through my handy @TerryE -to- Entry Level Application Developer translator results in:

"The current implementation of NodeMCU Docker Build (https://github.com/marcelstoer/docker-nodemcu-build) by @marcelstoer does not support LFS"

If my translator is working then I really think that statement should be included in the docs (until which time someone develops a working NodeMCU-Docker make script).

If my translator is broken then my original request for a NodeMCU-Docker Quick Start stands as-is.

Now off to pop a few Ibuprofen...

marcelstoer commented 6 years ago

I have to admit that I haven't tried myself yet but off the top of my head I don't see a reason why it shouldn't work. It's calling the same make as our CI build or my cloud builder - delegating all the hard work to this project's build scripts.

The only thing that is different is that you need to modify the various .h files yourself. (Some of) those build options are documented at https://nodemcu.readthedocs.io/en/latest/en/build/#build-options (LFS being a notable exception) and the LFS specifics are in the LFS page at https://nodemcu.readthedocs.io/en/latest/en/lfs/#selecting-the-firmware.

What would make things clearer for you (and others I guess)?

HHHartmann commented 6 years ago

@marcelstoer what @jmd13391 ist trying to say is that there is no documentation/script for Entry Level Application Developers. Some step by step like for building with Docker.

I supposed elsewhere to have some scripts which can be called inside docker and have the cmd.sh only print an explanation of what can be done. So just calling docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build would print the explanation which then would show commands like docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build build or docker run --rm -ti -v `pwd`:/opt/nodemcu-firmware marcelstoer/nodemcu-build lfs-image 'build' and 'lfs-image' being scripts which perform the action. @marcelstoer I could prepare a pull request if you like.

jmd13391 commented 6 years ago

@HHHartmann, Spot-On.

Guys (a.k.a. NodeMCU firmware developer gods), IMO the goal of all the countless hours of effort you have put in to make NodeMCU what it is today (phoenix from the 0.95 days) should be to make using the ESP platform via the lua/Node.js-like programming model as intuitive and easy to use / easy to ramp on to the ESP gettin' 'er done highway as possible. In order to reach that goal, you have to look at the documentation with eyes of a newbie application developer. Example: Close your eyes and visualize for a moment that you have absolutely no idea what a "make" file is or that you have no idea how to code in any of the C programming language variants.... words like "memalloc" are alien to you... (yes, I know, painful... keep your eyes closed). Flush everything from your brain except lua, Node.js and the BASICS of common working environments like Windows, Linux, Docker. Keep the documented modules of NodeMCU in there too. Now, open your eyes and describe the step by step process to get a xxxxx.lua application running using the LFS approach - using Cloud Build (under Win, Mac & Linux) & Docker. Keep in mind that your audience at this point is absolutely not interested in understanding the mechanics behind each step (that comes later when they level-up)... they are only interested in the steps themselves.

I don't care how a my washing machine cleans my clothes... I just want to know how to use the washing machine to get my clothes clean as quickly and easily as possible so I wouldn't be late or smelly for the date with my hot girlfriend.

Get the docs to that point and soon there will be a lot more coders out there building ESP applications using LFS NodeMCU then there are using the Arduino IDE. I look forward to the day when I Google search for an instructable or code example and get more NodeMCU hits than Arduino IDE hits. ;-)

-Joe

TerryE commented 6 years ago

I don't know Joe's developer background, so let me suggest a "typical" developer who is wanting to use NodeMCU:

This type of developer might want to pick a Lua environment over an Arduino C-based one. For this developer, using docker represents a real learning curve as everything is so foreign. Shit, I've used 4 mainframe OSes, ditto mini OSes, even more *nix variants, and everything from CP/M to all of the WinX variants and I still haven't bothered to get to grips with Docker, as for me standing up a base Linux VM is a 15 min job.

Going back to the cmd vs powershell and docker either way, if I we doing this I would want a BAT file or a function to wrap the docker run --rm -ti -vpwd:/opt/nodemcu-firmware bit so I can just do one of:

docrun help
docrun build
docrun lfs

that is the default directory should be pathed in. (and it shouldn't be marcelstoer anything) and a basic help should explain the commands (scripts) available and how you can tailor them.

What I just don't understand is why developers want to use docker to do luac.cross.

Surely docker is overkill in this second case. I don't understand why developers just don't use Cygwin as it takes less than 5 mins to download the Cywin setup, tick a few boxes, download the latest NodeMCU source, tweak user_config.h and run the app/lua/luac_cross make and now you have a binary that you can run directory on your Windows machine.

And for those developers than just want to stop at the "tick a few boxes" stage, here is the luac.cross.cygwin image to run in Cygwin, but remember to move it to a folder on your cygwin path and rename it to luac and do a chmod +x on it. (Its MD5 is 15e5010626d435ea48aaf1e64879d686). The only problem that I had was to stop Avast treating it as suspicious.

Yes, Gregor please feel free to document this and make it more developer friendly

HHHartmann commented 6 years ago

maybe it's just a matter of documentation and easyness why people start using docker and not cygwin or a linux VM. For docker there are just a couple of commands needed and provided to get a build.

As for setting up gygwin even "tick a few boxes" is easy if you know which ones. Otherwise you are just lost. Someone not knowing anything about linux/posix even does not know what "do a chmod +x on it" means.

I could have used cygwin or any linux but it was just easy to install docker and issue two commands to have a build done.

I agree that to go the full way, there should be batches to even wrap the somewhat long docker commands. But then again who should modify the path to include the batches. Do we also need some power shell scripts to automate this?

TerryE commented 6 years ago

Do we also need some power shell scripts to automate this?

IMO, if you've mastered power shell, then you are a long way into being able to use native WSH or Cygwin, If we are going minimalist then we are really talking about CMD prompt batch scripts, but yes having one person take on this learning curve for the community of WinX developers would be great.

jmd13391 commented 6 years ago

@HHHartmann Spot-On,,, again,

I will also add that there is a greater chance of someone new to NodeMCU & the ESP platform already having Docker installed... and very high probability of those people complaining about having to install yet another supporting environment (Cygwin) just to get work done.

I will speculate that the Cloud Build Service is the most popular method of getting from A to Z with the ESP & NodeMCU simply because it happens to be the path of least resistance. Unfortunately, there are several modules and features of NodeMCU that are not exposed on the Cloud Build Service. For application developers such as myself that want/need to leverage some of those "not enabled by default" functions and have no desire to dive into the bowels of the full NodeMCU toolchain, The Docker Build method has been the compromise... and the docs on how to build using Docker has been adequate... until LFS came along.

If using LFS (and I really, really need to use it) means that I have to stop application development, install and get comfortable with yet another build procedure (Cygwin), so be it. I've been staring at the LFS carrot dangling a few inches from my mouth for months now... (waiting for the paint to dry, trying to decode Terry's LFS whitepaper, and trying to find the build "path of least resistance". Time to accept that at least for the moment, I'll have to go the Cygwin route.

I'll leave you with this final comment: As someone who has been in this industry as long as @TerryE has (if not a bit longer), I can tell you without hesitation that being able to access LFS and the non-default features of NodeMCU via @marcelstoer's Cloud Build or Docker paths will be a major boost to the NodeMCU user base. LFS is a work of art... Make it super easy for the world to use and great things will follow.

-Joe

marcelstoer commented 6 years ago

Guys, can we please stop discussing the respective merits of the various ways to build NodeMCU - and focus on the issue(s)? I don't mind if other people like technology X, Y, Z better than Docker. It's big in the "cloud" and microservices architecture world and lots of developers have it already installed for that reason (may be their day jobs). Likewise, I'm not the least bit offended if people don't want to use my NodeMCU image; I don't take it personal. Back in the days I simply created it because it felt easy to do and offered an intermediary abstraction between the cloud builder and a Linux+toolchain environment. However, so far it's been pulled >9k times which means a few thousand individuals (I'm guessing 1-2k) find it useful. I just accept it as a fact and are humbled by that relative success.

That being said it's of course not so good that it couldn't become better. I'm happy to assess ideas filed as issues over at https://github.com/marcelstoer/docker-nodemcu-build.

TerryE commented 6 years ago

Marcel, we're just having fun :smile:

Gregor (@HHHartmann ) will sort out the scripts and he knows that he can use me if he does need any expert help on the build itself.

I'll put the luac.cross binaries on my ellisons,org.uk website for those that do want to use bare linux or cygwin and can't be bothered to do the builds, and I will do a simple webservice where you can post a ZIP of your source and get back an LFS image by return.

HHHartmann commented 6 years ago

@jmd13391 there are also a few steps to get an LFS image built with docker so your waiting is not too long. In recent dev there is a local/lua dictionary. Put your lua files in there and issue the following commands: Assuming that you have a float build or an integer build configured in user_config.h . Your build command docker run .... :/opt/nodemcu-firmware but appended by bash (there must be a space before bash) You will get a prompt like

root@56d64166f5e4:/opt/nodemcu-firmware#

enter the following three commands:

cd tools
make
exit

Ignore the error at the end of the build command. This generates the file local/fs/LFS.img

jmd13391 commented 6 years ago

I feel the point of this "issue" has been adequately expressed. Feedback has been provided to my satisfaction. The seed has been planted... I will close this issue. Thank you all.

-Joe

TerryE commented 6 years ago

@jmd13391 Joe, you are an active Lua developer who is interested enough to give constructive feedback. IMO, I would welcome you tracking other issues and giving feedback based on your own experience as a developer.

Thanks again :smile: Terry

HHHartmann commented 6 years ago

@jmd13391 my instructions above have been wrong. It must be make instead of build. Sorry for that. Corrected it above

You are also aware of Terrys online service and the new version of docker(not sure if it is live already)?

jmd13391 commented 6 years ago

@HHHartmann Thanks! I will give that a whirl...

I am aware of Terry's online luac.cross online service and plan to incorporate it into the quick-start doc "flow".

However, I'm not sure what you mean by "new version of docker" (other than the obvious "a docker update is available for download...")

-Joe

marcelstoer commented 6 years ago

I guess he means the LFS-image building feature in my NodeMCU Docker image that landed last night thanks to Gregor's hard work.

poorandunlucky commented 5 years ago

I don't want to create a new issue, but there's really 0 documentation on the internet regarding the LFS... I think a lot of work probably went into it, and compared to other features, it might be largely underused because, as the original author wrote, it's a bit "unintuitive" to the new user...

The documentation page is great, and it was a great read, and I'm really happy to be part of the NodeMCU family, but I still don't really know what the LFC can do for me in real life, and it's a bit unclear to me at this point why the option to create a LFS is available on the nodemcu-builder.com website if I can only use the LFS by compiling my firmware using the Linux toolchain, or the Docker image...

Maybe a few examples, and a few use cases would be useful? A demo, of some sort? I have nice concepts in my head, but I'm still very abstract on how to leverage the LFS...

Seriously, though... Searching for "nodemcu lua flash store" on Bing yielded three relevant results, two being the official documentation page, and the other this page. It's like nobody ever used it, it's not part of anyone's workflow, nobody talked about it, either... no news, no blog posts... nothing. I really don't think people understood what it does, because it sounds really great, especially for web-enabled applications or interfaces with a lot of static text, and if it's also useful where there's dynamic text as well, I'd sure like to know because "double your RAM" is an easy selling point!

I'm really amazed nobody talked more about this, seems like big news in the ESP world, and my only guess as to why is that nobody understood how it works...

If I'm wrong, and if there's a good example you can think of off the top of your head, I'd appreciate that, in the interim... : )

nwf commented 5 years ago

@poorandunlucky LFS is not a generic "double your RAM because you turned it on" kind of thing. The LFS holds compiled lua files, which will be specific to your project, much as the SPIFFS holds lua files.

All that said, https://github.com/cmukgb/ctfws-timer-iot/blob/master/mklfs.sh might be of interest: this is how that project builds its LFS image. https://github.com/cmukgb/ctfws-timer-iot/blob/master/mkspiffs.sh lands the LFS image in the SPIFFS image that gets flashed down, and https://github.com/cmukgb/ctfws-timer-iot/blob/master/init2.lua knows to update LFS whenever it has a luac.out file in SPIFFS. So, having been flashed, the system will land the LFS image and reboot. If you want to turn this flow into documentation, please open a PR. :)

TerryE commented 5 years ago

I still don't really know what the LFS can do for me in real life

If you use it you go from 44 Kb RAM for Lua code and data to upto 256KB for Lua code and read-only data and 44Kb for application data.

it's a bit unclear to me at this point why the option to create a LFS is available on the nodemcu-builder.com website if I can only use the LFS by compiling my firmware using the Linux toolchain, or the Docker image...

Read the whitepaper in the docs if you want to understand the details. The firmware builds a binary to run on the ESP that is LFS-enabled. You still need to compile Lua code on the dev machine into an LFS image. At the moment you still need to do this compile of the compiler yourself. On Windows, you've got a choice of four toolchains to do this: WSL, Cygwin, MinGW and native MSVS. Your pick.

Sorry about Bing. I don't use it. And I only develop the Lua core stuff; it's really down to others to help evangelise it.