Question: Why is there a copy of grbl in each driver?

[mnesarco]

As far as I understand, the main goal of grblHAL is to have a one common grbl core and many HAL drivers (one per hardware variation). But i can see that each driver folder have its own copy of grbl so i am a bit confused about the architecture. If i want to add a new processor driver do i need to copy the whole grbl inside the driver folder too?

[phil-barrett]

It has to do with the limitations of github. Yes, you copy the contents of the main grbl directory into src/grbl in each driver tree you are building from.

[kdomanski]

How about using symlinks? I'm not a Windows user, so I cannot test how Github Desktop behaves, but I replaced the ESP32 copy of grbl with a symlink in my branch https://github.com/kdomanski/grblHAL/tree/symlink and it builds just fine. From what I gather, symbolic links should in principle be supported on Windows.

[dresco]

How about using symlinks?

fwiw, I ran into a couple of issues using symlinks when building the Teensy driver with PlatformIO. Fixable, but was due to plugins using the Arduino framework having "../blaah.h" includes, and relative paths being resolved relative to the destination of the symlink.

There is obviously duplication of the core files, but at least setup is easier now all the per-driver grbl & plugin folders are populated automatically from Terje's SVN source.

[mnesarco]

Usage of git submodules is a better solution.

To achieve that:

1. grbl32 core must be a repository
2. Each driver must be a reporitory
3. Each driver repository configures a git submodule pointing to a specific version on grbl32 core repo.

https://git-scm.com/book/en/v2/Git-Tools-Submodules

[terjeio]

Submodules are coming here? I will at least do a test to see if it is a workable approach.

I use symlinks for Arduino development in the Eclipse IDE (on Windows), sadly it is not possible to check out symlinks...

I keep a local master in subversion, for that I use externals that fits this project perfectly. Sadly there is no git equivalent.

[mnesarco]

git submodules were designed exacly for this usecase: using extenal repositories inside other repositories keeping independent versioning. submodules does not require symlinks, are portable, and track versioning very well.

https://github.blog/2016-02-01-working-with-submodules/

There are no restrictions on the location of the repositories, so grbl32 core and drivers can be located anywhere.

[kdomanski]

IMO the previous setup (requiring code copying) was far less confusing for users.

sadly it is not possible to check out symlinks...

You mean in SVN? AFAIK it's a Windows-only limitation of Subversion. :/

[mnesarco]

@kdomanski , git submodules does not require symlinks. IMO Copying things in multiple parts is a maintenance nightmare.

[terjeio]

sadly it is not possible to check out symlinks...

You mean in SVN? AFAIK it's a Windows-only limitation of Subversion. :/

Not SVN, github. Symlinks are not needed with Subversion as externals takes care of fetching the required code. I use symlinks to be able to use an real IDE instead of a toy IDE for Arduino drivers. I still use the Arduino IDE for compiling/uploading since the boards are not debuggable (except the Due?). I can live with that, but I am not going to add more non-debuggable drivers to the family as they are far too expensive to work with.

IMO Copying things in multiple parts is a maintenance nightmare.

So git submodules are bad too? The setup I have now is easy for me to maintain as I have a staging folder that I update from subversion before I push to github. The downside is that I have to keep the subversion repository in the background?

Here is how the iMXRT1062 (Teensy 4) driver is organized in subversion:

[mnesarco]

You can do the same with git submodules.

• Each dependency (module) can be an independent repo with its own version, issues, pull requests, etc...
• You can import any repo (external in svn terms) into any other repo.

Each submodule behaves as a local sub folder copy, but it is managed by git.

So a driver could be a repo with very few own files, and a set of imported submodules pointing to external repositories. In the filesystem it will be exactly the same as before: a folder with subfolders. But all the management will be done by git.

[raumneun]

+1 for submodules. We use it very regularly at work with great success after you got your head around it. It would also have the benefit that one could keep track of relevant driver changes more easily. Especially when new drivers are developed, I read through all commits just to see if I need to update or not. With submodules, I would just subscribe to the core, the plugins I use and the driver I need. Of course, this is "just" convenience, but the structure of submodules is perfect for this case imho.

[terjeio]

A negative with submodules is that the container directories are empty if dowloading a zip file. Any known workaround for this?

[mnesarco]

@terjeio yes, submodules requires git because submodules is a development feature not a distribution feature.

[terjeio]

yes, submodules requires git because submodules is a development feature not a distribution feature.

So what to use for distribution? Set up a website for that as most users are not developers?

[mnesarco]

It depends on your vision of the project. What I see is a set of libraries that can be combined to create a firmware, it sounds like a product for developers not a product for final users.

This project is amazingly useful and I see it will get a lot of attention and will grow pretty fast and things will start to be wild. So you should make things easy to maintain for your own benefit.

Any "user" capable of combine c/c++ libraries, compile and flash a firmware into a MCU should be capable of use git.

[raumneun]

Any "user" capable of combine c/c++ libraries, compile and flash a firmware into a MCU should be capable of use git.

I would second that argument. Alternatively, if people insist to not use git (which even has nice GUI clients like SourceTree, Gitkraken, ... for those intimidated by the command line), they can still download the zip files of the plugins and the core and extract them in the respective folders. That is not (much) more work than copying the files as it was before and would greatly simplify maintaining the code and finding more people to contribute and develop drivers.

Alternatively, you could use a post-receive hook (or maybe github has something like CI pipelines, we use gitlab for a similar purpose) that creates zip releases from tags and pushes them directly into the driver repository. But to be honest: if this is to satisfy users that have never compiled anything, I think a complete build pipeline that provides ready to flash images would be more appropriate. Of course also more work, but maybe this is something to leave as an exercise for the community. I guess @mnesarco is right, it depends on the vision you have for this project and if you want to focus on development or if you want to provide a simple way for the "average" maker that wants a turnkey system.

[terjeio]

I'll keep both repositories (this and grblHAL) active for a while and see how it pans out. This since it is fairly easy for me to keep both in sync from the master subversion repository - I am not letting that go.

PRs for this will no longer be accepted (I guess I can block that) when the new is online - perhaps tomorrow.

But to be honest: if this is to satisfy users that have never compiled anything, I think a complete build pipeline that provides ready to flash images would be more appropriate.

I would love to see this - but I do not have the time to do it myself.

... if you want to focus on development or if you want to provide a simple way for the "average" maker that wants a turnkey system.

IMO both developers and makers needs should be taken care of - without users why develop at all? Or should I leave it to other developers or system integrators, such as @phil-barrett, to provide turnkey systems for users?

[kdomanski]

It depends on your vision of the project. What I see is a set of libraries that can be combined to create a firmware, it sounds like a product for developers not a product for final users.

Second that. In addition, isolating non-devs a bit from this repo would prevent the issues tab turning into a support forum.

[raumneun]

I would love to see this - but I do not have the time to do it myself.

I guess https://docs.platformio.org/en/latest/integration/ci/github-actions.html would be the way to go, however, I have never tried that myself. One issue is that even though the build chain might be somewhat straightforward to set up, driver level configurations such as pin mappings or which plugins to use would be less simple as it requires a user to somehow interact with the build chain and "order" a flashable file. I guess it is possible but not worth the effort and I would agree with your previous statement:

Or should I leave it to other developers or system integrators, such as @phil-barrett, to provide turnkey systems for users?

I guess most people will use grblHAL for their DIY machines. The thing I love about developing a CNC is the many skills you need to achieve your goal: mechanics, electronics, software, hardware, it all comes together. But my experience is that "makers" typically come from one field and use the CNC project to gain knowledge in the other fields, especially electronics. That's why people buy Eding-CNC, Mach3, EstlCam, .... or plain grbl shields for Arduinos. I guess my point is that people that develop their own board will be able to use git with submodules. People that hesitate to develop their own hardware and want a turnkey system will buy something like Phil's board and look for a working firmware with different settings (number of axes, enabled plugins,....) at the project page of the breakout board. If they later on want to dig into coding or compile with custom settings, they can start off by downloading the zip files and assembling a folder structure manually (could be described in a wiki page or the README.MD of the driver repository or on readthedocs.io ,...., I would be happy to provide a short article for this).

[phil-barrett]

I think it important to point out how important the end users are for grblHAL to be successful. If it is just a set of libraries for developers, it will send the end users elsewhere. Looking at how Grbl became the most used CNC software is instructive. It was possible for an end user to download and build with no programming skills at all - just simple instructions on what to do to get a copy of Grbl loaded in their Arduino. Right now, many users are able to download the zip file and build for the Teensy. You all don't see the support I provide for that which is around 4 times what happens here.

I agree that a Marlin style configurator (is that even a word?) would be great but that isn't going to happen soon. I would be happy to support efforts in that direction and am willing to fund server space but don't have the skills to do the actual work. The build interface would be available on grbl.org. I would like to see the iMXRT 1062 and STM32 targets first as those are the most popular. After that I'm not sure the order of the next ones.

[mnesarco]

I can create a simple workflow in GitHub Actions to automatically create and publish a zip with submodules included. Autoexecuted on push events.

[phil-barrett]

Hmmm, that sound like it has potential. Would it be possible to allow user selection of the options available in my_machine and and the corresponding map file?

The key is a simple process for the average (ie non-programmer) user to get to a file they can upload to their microcontroller.

[mnesarco]

Hmmm, no. GitHub Actions are automatic tasks executed on repository events. They are not user GUIs. It can solve the problem of providing a zip file with everything inside as before submodules. The configurator idea is a user application, another kind of thing.

[mnesarco]

Here is an example repository with a submodule pointing to grblHAL/core and with automatic generation of a downloadable sources.zip file in Releases Section containing the submodules:

https://github.com/mnesarco/actions-test3

The workflow (Github action is a simple file inside the repo): https://github.com/mnesarco/actions-test3/blob/main/.github/workflows/sources.yml

The sources.zip file is generated on any push with a tag starting with letter 'v'. ie v1.0.1

[terjeio]

@mnesarco Thanks for providing the action.

The sources.zip file is generated on any push with a tag starting with letter 'v'. ie v1.0.1

A push on the driver repo I assume? If there are core or plugin changes with no driver changes then what? Or am I missing something?

[mnesarco]

Yes, a push on the driver repo. A script to get the latest submodules can be written, but it is usually a bad idea. Suppose you made a breaking change in the core that works only with some drivers, then the old drivers must point to the previous version until they update to become compatible. The driver developer decides when to update its dependencies, same applies for plugins. This is not a disadvantage, but on the contrary, this is a feature. Each driver can be safely developed at is own pace. At this point, apparenty you are maintaining all drivers and plugins by yourself, but with this architecture, anyone can start a new driver anywhere and maintain it by itself, so your grblHAL-core is a product by its own with a much wider scope. The same applies for plugins, anyone can create plugins anywhere.

Now suppose you are a driver developer, you have only your driver files and a bunch of submodules pointing to the versions that works with your code and it will never break due to core/plugins changes in another place.

[phomann]

Hi all, Splitting the repository into individual repositories seems like a good way to partition the various modules and avoid. duplication.

Currently I'm using the Arduino environment to build for Teensy a 4.1 platform, and am coping the relevant folders into the Arduino library folder. Can someone provide a step/step guide for using the new repository on how to extract the code (create a zip) from GIT, which I presume that I then copy to the Arduino library folder. It would be great if platformIO could be used.

Cheers,

Peter

[dresco]

Can someone provide a step/step guide for using the new repository on how to extract the code (create a zip) from GIT, which I presume that I then copy to the Arduino library folder. It would be great if platformIO could be used.

@terjeio I've updated my PlatformIO Teensy build instructions for the new repository structure, I can either send them to you to update the new wiki, or if you can give me access I'll update myself? Cheers..

[terjeio]

Can someone provide a step/step guide for using the new repository on how to extract the code (create a zip) from GIT, which I presume that I then copy to the Arduino library folder.

I'll come up with a guide soon - but have to look into how to handle zip-files. I did not immediately like the git action proposed by @mnesarco as I think this will be hard to maintain. I'll have to learn more about submodule usage/options before I decide on a path forward...

Currently the best option is to use git or github desktop to download as these will download the dependent submodules automatically.

@dresco Will your PlatformIO build instructions be relevant for other drivers and thus should be added to the core Wiki, or should it be added to a driver specific Wiki? Or even to a webpage covering the whole site?

[phomann]

Sorry, the zip-file inclusion was a bit of a red herring. I was just using it as an example. I have managed to get it to work using tortoise git as suggested. Cheers, Peter

[dresco]

@dresco Will your PlatformIO build instructions be relevant for other drivers and thus should be added to the core Wiki, or should it be added to a driver specific Wiki? Or even to a webpage covering the whole site?

Good point, instructions are driver specific, though many principles would be common between drivers..

Is it a concern that information may get more fragmented if split across the various driver repos? Not that I have a good answer for that, other than put the Wiki in a separate documentation repo perhaps?

[langwadt]

what happens with submodules when you do a fork?

[terjeio]

It seems they are imported but will stay at the revision of the initial fork unless later pulls are explicitly told to update them?

git pull --recurse-submodules