search

machinekit / machinekit-cnc

CNC stack split out into a separate package
Other
60 stars 36 forks source link

UUID in machinekit.ini is always the same #39

Open ArcEye opened 6 years ago

ArcEye commented 6 years ago

Issue by machinekoder Fri Mar 13 13:00:36 2015 Originally opened as https://github.com/machinekit/machinekit/issues/531

The Machinekit Debian packages feature a default machinekit.ini file. There is an UUID values in this configuration file that is relevant to identify different Machinekit instances on the network. This UUID should be autogenerated after package install to ensure a different identification for every machine.

ArcEye commented 6 years ago

Comment by RunningLight Thu Mar 19 18:33:31 2015

Since some of us like to clone from one VM instance to another and otherwise reuse an installed Machinekit, it would be nice to provide a script which invokes uuidgen and then seds the result into machinekit.ini

The script then could be used during in the postinst phase and again later.

ArcEye commented 6 years ago

Comment by mhaberler Thu Mar 19 19:07:27 2015

could you cook up a PR?

ArcEye commented 6 years ago

Comment by RunningLight Thu Mar 19 19:37:15 2015

I'll give it a shot.

ArcEye commented 6 years ago

Comment by mhaberler Fri Mar 20 05:24:28 2015

I think it boils down to intelligently deciding which file to fix:

on a packaged install there's no 'make something', so probably adding a script to fix up the global machinekit.ini is the way to do it

in a source build, a make target might be more appropriate, so the result is permanent if user decides she wants a 'make install' eventually.

it's a late fallout of our forebear's singletonitis - what do you mean, instances, we do no stinkin' instances around here ;)

cerna commented 5 years ago

While investigating machinekit/machinekit-hal#170 issue I discovered that this issue probably never was addresses and there still are Machinekit Instances running with GUID of a42c8c6b-4025-4f83-ba28-dad21114744a which would create major problem in my code as I was under impression that this identifier is unique. In other words it is very bad for me.

So I decided that I will correct this issue. Looking at the original machinekit/machinekit#531 there was idea by @mhaberler to solve this in two stages. For compatibility I propose to solve this in three stages:

  1. The current used GUID MKUUID=a42c8c6b-4025-4f83-ba28-dad21114744a in scripts/machinekit.ini.tmpl.in file will be declared as GUID non grata and no Machinekit Instance should be run with this identifier. This should be documented in the file scripts/machinekit.ini.tmpl.in in comment to explain it to every user.

  2. As I am not so great with bash scripting and I have discovered no great libraries for parsing and writing of INI files in linux bash script I would add crudini to Debian package dependency of Machinekit and update manual for build from source (because of mk-build-deps probably no change will be needed).

  3. In the debian/machinekit.postinst file I would add section that on new install of Machinekit would create new GUID and with use of crudini will update the machinekit.ini file. On upgrade from previous version of Machinekit will query if current GUID in .ini is not GUID non grata anf if not will do nothing, on positive will create new GUID, create backup of the file and change GUID for new one.

  4. On build from source will just generate new GUID and change it.

  5. To filter out old machinekit.ini files used in examples and specific configurations, I would also need to check the GUID before each run of Machinekit. Am I correct that the file scripts/realtime.in is accessed with every Machinekit run? If so then in this script check if the GUID of passed MACHINEKIT_INI is GUID non grata and if so create backup of the passed ini file, generate new GUID and save it in passed file

This should guarantee that GUID of the running Instance is always unique. This will not solve @RunningLight problem of cloning VMs, but this is so fringe case as Machinekit is not intended to run that everybody doing it should be covering this in his own solution. I also would not consider @cdsteinkuehler solution of separating GUID to it's own file to be particularly pertinent solution.

Did I miss something? Will it break anything?

ArcEye commented 5 years ago

I think the simple solution is just to do what @RunningLight proposed and ensure that it covers both package and RIP builds.

Then when you upgrade, all UUIDs will be unique.

IMHO your approach is too complicated and adding another dependency so as not to use an available command line tool, is not a good idea.

Neither is blacklisting the existing UUID and relying on users to read something, they won't. If you want to use unique UUIDs, you upgrade to the package version that supports it. If you don't, just carry on as you are.

The reason the UUID being the same has only ever been a hypothetical problem, is that 99.99% of users do not use them. The original idea behind it all was an aspiration of capability for its own sake, which has few useful applications.

dkhughes commented 5 years ago

The way I handled this is through a systemd startup script, but I have the advantage that it was targeting an image based distro (arm for zynq). It's a simple bash script that looks to see if a UUID has been generated, and replaces the entry in the /etc/linuxcnc/machinekit config file. Adding the meat of that UUID generation to a new script in the scripts folder of the machinekit install would be easy. Then we can invoke it from the startup script for images based on omap-image-builder, or by hand for manually installed packages. I'll convert it and make a PR.

cerna commented 5 years ago

The reason the UUID being the same has only ever been a hypothetical problem, is that 99.99% of users do not use them. The original idea behind it all was an aspiration of capability for its own sake, which has few useful applications.

That's because majority of user base only wants LinuxCNC running on BeagleBone Black and other single board computers and does not care about special middleware features because what they are interested in is user-friendly user interface which hides almost everything (like the Tormach's PathPilot).

When you try to use UUID of running Machinekit Instance for identification of objects representing remote machinekits and pass them triplets of PTR-SRV-TXT DNS resource records again based on this identifier in your implementation of Service Discovery, it becomes problem very quickly. (All because you are under impression that "unique" really means "existing as the only one or as the sole example".)

IMHO your approach is too complicated and adding another dependency so as not to use an available command line tool, is not a good idea.

Could be. I just couldn't find anything else which would offer such ease of use and was activelly maintained by someone else and was adhering to ini file standard (multiple sections possible with same keys etc). (So active, so with hopefully prompt removal of bugs.)

Neither is blacklisting the existing UUID and relying on users to read something, they won't. If you want to use unique UUIDs, you upgrade to the package version that supports it. If you don't, just carry on as you are.

I didn't want to make user life dificult. If they did changed theirs GUID, the nothing would happen, if they did not, they probably don't care enought, so automatical change would not bother them much. The only point of liaison would be if they defined specific .ini file (for example in examples and tutorials) and then script changing GUID from default would ask for sudo password. Everything else would be automagicall.

I think the simple solution is just to do what @RunningLight proposed and ensure that it covers both package and RIP builds.

I thought that it is just this. Just little more refined and taking in the fringe cases.

The way I handled this is through a systemd startup script, but I have the advantage that it was targeting an image based distro (arm for zynq). It's a simple bash script that looks to see if a UUID has been generated, and replaces the entry in the /etc/linuxcnc/machinekit config file.

Doing something like 

# Update to new MKUUID in the /etc/linuxcnc/machinekit.ini file

inifile=/etc/linuxcnc/machinekit.ini

set_new_uuid() {
        uuid=$(cat /proc/sys/kernel/random/uuid)
        crudini --set --existing $inifile MACHINEKIT MKUUID $uuid
}

if [ "$1" = configure -a -z "$2" ]; then
        set_new_uuid
elif [ "$1" = configure ]; then
        if [ $(crudini --get $inifile MACHINEKIT MKUUID) = "a42c8c6b-4025-4f83-ba28-dad21114744a" ]; then
                cp -p $inifile ${inifile}_backup_$(date +%Y-%m-%d_%H.%M.%S)
                set_new_uuid
        fi
fi

in postinst is simplest solution for package install, I think.

dkhughes commented 5 years ago

Since we control the ini file, and enforce the syntax, you can use a simple sed to replace the uuid in place:

set_new_uuid() {
    sed -i "s|^MKUUID=.*|MKUUID=$UUID|" /path/to/target/machinekit.ini
}

if they did not, they probably don't care enought, so automatical change would not bother them much

Only possible problem here is I think it could affect end users. I cache the UUID in remote interface tools, and I think @machinekoder 's Qt based GUI's do as well. So, after package update their machine might not connect automatically.

The implementation I have been kicking around allows the script to be passed a uuid value through command line arguments. This means someone could keep a UUID in a separate file (I do this since the uuid is generated from a timestamp and MAC address and we want it permanent for the life of the image), and simply cat the value in as an argument, or use uuidgen if it's available, or cat the /proc filesystem like in your example, e.g.,:

update_mkuuid -f /etc/linuxcnc/machinekit.ini $(cat /path/to/uuid_file)

or

update_mkuuid -f ~/rip/dir/etc/linuxcnc/machinekit.ini `uuidgen`

Adding a make target for RIP builds allows developers to benefit if they need unique UUIDs (like me). I'm clumsy with makefiles, but I was basing it off similar logic to the setuid target.

ArcEye commented 5 years ago

I have been tracing through the current situation, so as to get it straight in my head. I can now see why your solution got so convoluted @cerna :laughing:

Just restricting ourselves to RIP builds for now.

configure.ac checks the uuidgen is installed and will halt without it. Then it generates a new UUID

checking for uuidgen... /usr/bin/uuidgen
setting unique Machinekit UUID to 41360910-7f3a-4efb-8fa4-488d3f9b0a16

If you commit a change, you will be prompted to run autogen.sh / configure again Doing so will generate a new UUID, which because it is not used, does not matter currently.

It then checks if a BUILD_TOP_LEVEL machinekit.ini is installed in $BUILD_TOP_LEVEL/etc/linuxcnc If not it copies machinekit.ini.tmpl to $BUILD_TOP_LEVEL/etc/linuxcnc/machinekit.ini This prevents overwriting a RIP UUID that has been changed by the user in a rebuild.

That will survive everything but a git clean -xdf (which you might use to clear all the fluff before making a commit.)

It does NOT check if a system toplevel machinekit.ini exists, thus allowing a user to maintain the same UUID across RIP builds

It also does not use the new uuid and does not allow a global ones use, even if it does exist From rip-environment.in:

MACHINEKIT_INI=@EMC2_HOME@/etc/linuxcnc/machinekit.ini; export MACHINEKIT_INI

ie. restricted to @EMC2_HOME@, which is RIP/etc

Scripting the reading of UUID and writing the new one is the easy part, getting the sequence right so that existing valid UUIDs are not removed needs some more thought.

The easiest path is probably to use @cdsteinkuehler s idea of separate MKUUID files

Does that sound rational so far?

Packaging, to my mind at least, is less problematic and can be dealt with separately once RIPs are sorted

dkhughes commented 5 years ago

I think your ideas sound fine, but I've had a thought.

Can we circumvent this problem in both RIP and package installs by specifying a path to the UUID we want to use instead of the value itself? If the file doesn't exist, just let the code default to the current MKUUID in machinekit.ini. Basically, add an option for MKUUID_PATH? It would be backwards compatible, nothing in the git tree to worry about, and if an install does have a file at the path specified, we just use it.

ArcEye commented 5 years ago

I have written a script which is called from configure and will do what I outlined above ie. parse the MKUUID from the system files and use that MKUUID if it is unique or the generated one. It can also update the system files to the generated one and create an /etc/linuxcnc/mkuuid to hold it, either from the existing unique UUID or the generated one. Just tested and it works fine

Where do you want to specify MKUUID path? In a switch to configure? Or do you mean replace the MKUUID= line in the ini file with a path and also specify a FALLBACK_MKUUID= in the .ini file, which can be the generated UUID.

dkhughes commented 5 years ago

replace the MKUUID= line in the ini file with a path and also specify a FALLBACK_MKUUID= in the .ini file

Yes, exactly. As an additional option. I wouldn't rename the MKUUID variable though for reasons below. In the template, a comment instruction could say that a valid path overrides the legacy MKUUID value. Then we can add a make target that the end of compile help text can offer, similar to the required security settings or setuid target. Something like:

"If you need to generate a unique ID, run sudo make gen_uuid"

I agree that end users will skip that if it's in the docs, but someone custom compiling a RIP build should see that message. The new target would create and install the /etc/linuxcnc/mkuuid file.

In the machinekit.ini we have the extra MKUUID_PATH=/etc/linuxcnc/mkuuid. In the mklauncher, mk_service* and friends, we check if a file with a valid UUID exists at the path specified and use it, or we fallback to the MKUUID value that is already specified (Backwards compatible with existing ini files in service).

For package installs:

I think is the least amount of impact on existing installation or RIP builds, and no git tree worries. What do you think?

dkhughes commented 5 years ago

Oh, and if for some reason someone wanted multiple RIP builds with unique UUIDs on one machine, they could edit that MKUUID_PATH value for each rip configuration - a free side effect I think.

cerna commented 5 years ago

I am not a fan of settings strewn across multiple files as that creates a potential for error by omission. I think that using the power of INI file sections would be a better solution. But if you think that this solution is cleaner, better or simpler then who am I to argue. Right? ffff

Since we control the ini file, and enforce the syntax, you can use a simple sed to replace the uuid in place:

Still feels like a hack. I am not saying that it doesn't work, just that it works on a subset of INI files and requires special considerations on changes in regards to INI standardization. I just don't like these types of solution. That's all.

Only possible problem here is I think it could affect end users. I cache the UUID in remote interface tools, and I think @machinekoder 's Qt based GUI's do as well. So, after package update their machine might not connect automatically.

From that perspective, it could. But it cannot be helped. They are using a non-unique unique identifier. It's a one-off change which will solve a crippling issue from a remote communication point of view. I don't know what is @machinekoder doing in his Qt programme. I tried to study it, but in the end, I decided that I don't have the time to study yet another programming language and framework. But your usage doesn't seem quite typical in Machinekit's terms. And you are using "unique" GUIDs, so nothing should change for you or others like you.

This means someone could keep a UUID in a separate file (I do this since the uuid is generated from a timestamp and MAC address and we want it permanent for the life of the image), and simply cat the value in as an argument, or use uuidgen if it's available, or cat the /proc filesystem like in your example

You mean that you have some arbitrary file in your system which contains plaintext formatted GUID in it's hyphenated or unhyphenated form and nothing else, like a42c8c6b-4025-4f83-ba28-dad21114744a? Sorry, but that sounds like stuff from which nightmares are made to me. It's a matter of opinion, surely, but every file (more so in open-source) should contain some form of commentary or such. And some normalized form when it contains settings information is also nice (XML, JSON, INI, ...).

(...) benefit if they need unique UUIDs (like me). I'm clumsy with makefiles, but I was basing it off similar logic (...)

Again, a matter of opinion, but I think every GUID should be "unique" regardless of actual need.

I can now see why your solution got so convoluted @cerna

Glad my effort was not completely random or useless. Looks more complicated than I originally thought.

(...) uuidgen is installed and will halt without it (...)

Is this package really necessary? Is'n the cat /proc/sys/kernel/random/uuid enough? It!s not for cryptography or anything. Just for identification.

Otherwise, it does sound rational.

EDIT:

(...) If /etc/linuxcnc/machinekit.ini exists, we don't touch it. (...) and (...) Neither is blacklisting the existing UUID (...)

I would like for the current default GUID (as named by @ArcEye ) or GUID-non-grata as a name by me to be completely purged on new versions of Machinekit (package or RIP build) otherwise there will be no guarantee for uniqueness. And then I or anybody else when running Service Discovery could just drop resources with GUID-non-grata identifier.

BTW, shouldn't the INI file as the configuration resource be authoritative and from this in effect static? IE every change to it should be committed by external programme writing new value key part or changing an existing one. The path section to some external GUID seems little like leading programme through a rabbit hole. Frankly more I think about it reminiscend to me the problem of cyclical references in two classes having reference to each other (dependent and dependee, like collection item having reference to a collection itself) in GC languages and that is never a problem-free solution. Just saying.

dkhughes commented 5 years ago

GUID should be "unique" regardless

I think this is just semantics. If they're only running one instance in their domain it's unique enough for the application. I see a real issue with the hardcoded UUID in certain cases which is why I want to help come up with a better solution, but blacklisting the old UUID is heavy handed for people running one machine (3d printer, router, whatever) on a network.

arbitrary file in your system which contains plaintext formatted GUID in it's hyphenated or unhyphenated form and nothing else

No, I never said that. It could be an ini file format and we can continue to use the python ini file parsing structures that we already use... no extra dependencies.

Splitting the UUID into it's own file eliminates a lot of the issues I see with keeping it in one file. If there are multiple applications that want a unique ID running on a host, that file could even be entirely external given a path option in the main config file as long as it follows whatever format we choose (again ini makes sense considering that's used already).

Randomly generating a UUID on every package install, arbitrarily choosing when to change the UUID without end user consent, or managing a separate mechanism for RIP builds just sounds more difficult without any benefit over the minor inconvenience (is it really an inconvenience?) of 3 config files instead of 2 in /etc/linuxcnc folder.

cerna commented 5 years ago

I think this is just semantics. If they're only running one instance in their domain it's unique enough for the application. I see a real issue with the hardcoded UUID in certain cases which is why I want to help come up with a better solution, but blacklisting the old UUID is heavy handed for people running one machine (3d printer, router, whatever) on a network.

That's true. It's just impossible (or not impossible, but certainly more convoluted) from a remote point of view to identify one machine running with default GUID which is unique to a specific domain and multiple machines running with the same default GUID - more so as we know that we can have same hostnames or FQDM in local network.

No, I never said that. It could be an ini file format and we can continue to use the python ini file parsing structures that we already use... no extra dependencies.

In that case, I am sorry, I was under the erroneous impression that was the case.

Randomly generating a UUID on every package install, arbitrarily choosing when to change the UUID without end-user consent, or managing a separate mechanism for RIP builds just sounds more difficult without any benefit (...)

Maybe we are just looking at the usefulness of GUID differently. I am thinking about the uniqueness of GUID in terms of IP uniqueness and ability to identify an interface. I am not thinking about GUID as it would be Domain name, something which "name". For me, GUID is something unique at the moment, not something which necessarily stays constant across time.

dkhughes commented 5 years ago

GUID is something unique at the moment, not something which necessarily stays constant across time

Ah, that is a big difference in how I see it. I'm looking at the GUID as a unique key identifying a machine or image install. Long term, I would like to add the ability to machinekit to load configurations remotely. A unique fingerprint allows the remote GUI to know what machine we're talking to, and then follow that to a proper configuration file for upload.

This is a piece in the puzzle to address https://github.com/machinekit/machinekit/issues/104. Then, you could even have multiple machines able to run from a single client (3d printer cell perhaps), while maintaining a central location for machine configs.

cerna commented 5 years ago

Sure, you are looking at it like it was S/N. And I am certain that if we sometime will get to the authentication/authorization problem, it will be needed to use it this way. Or the Machinekit multi-instance machine communicating with components by way of haltalk.

However, everything has TTL. Even if the TTL is "until I manually delete this row from database".

There is also the question of exactly what GUID identifies? Certainly not configuration. And if not configuration, then probably not even hardware. (Special in form of Ethercat slaves or Mase hardware or classic PC hardware.) So probably just installation. And installation since when? Since package install? Since upgrade? Build? Actually, it is just string which does not mean anything by itself. Our dependant application gives it meaning. I am trying to build communication middleware to Machinekit Instances (something like QT Machinetalk bindings), so for me, the moment the last service on an instance identified by GUID dies, the instance dies and I care about the GUID no more. If I will be implementing some other software (for example based on this middleware) which will need to identify and remember Instance, I will be addressing it then.

For now, there is a need to implement a set of rules and based on that a solution, which will not be a kind of hack, will solve all our current needs so we all are happy and will be open for an extension on problems we are not realizing now.

This is a piece in the puzzle to address machinekit/machinekit#104. Then, you could even have multiple machines able to run from a single client (3d printer cell perhaps), while maintaining a central location for machine configs.

Yeah, my head hurts now with this problem and I am a strong proponent for small steps. (Or we won't get anywhere.)

ArcEye commented 5 years ago

I need to digest the above.

The ini file thing is a red herring really. Nothing in machinekit 'writes' to ini files, just reads them and there is a library for that.

The substitution of one key for another is just simple grep, parse and sed, couple of lines and easily done.

What I want to avoid is having to alter all the code that currently reads machinekit.ini to do something else. The key should be where it has always been, then we don't break anything we don't know about that relies upon it and keep things simple. How it gets there and in what circumstances is down to our ingenuity.

ArcEye commented 5 years ago

@dkhughes

For package installs: If /etc/linuxcnc/machinekit.ini exists, we don't touch it. By defaulting the value of the MKUUID_PATH to the value above, the check for the file should fail, and we don't alter an existing installation's UUID that remote interfaces could have cached. If /etc/linuxcnc/machinekit.ini doesn't exist, we install the new machinekit.ini.in template. Check if the /etc/linuxcnc/mkuuid file exists already. No UUID file? Generate a new one and drop it in the file. I think is the least amount of impact on existing installation or RIP builds, and no git tree worries. What do you think?

I was leaving package installs for now, but I generally agree.

However if /etc/linuxcnc/machinekit.ini exists at package install, the install will fail because it should have been removed and clashes with the file in the package.

So we will always need to install a /etc/linuxcnc/machinekit.ini , what MKUUID is in it is another matter. This is why a separate file for the UUID is important.

I think the install must check the default MKUUID_PATH. This will enable a user to maintain their particular UUID between upgrades. That file will not exist unless they have moved to the 'new system'.

We do not need to check anywhere else. The default path should always be /etc/linuxcnc/mkuuid If the user wants to use a rotation of keys, different one for each day, keep the key on a USB stick or whatever, that is up to them. They just make this a symlink which points wherever they want it to. Keep it simple.

                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Your notion of several clients on one machine contacting a remote server, each with a unique MKUUID, throws a bit of a spanner in the works.

If they were RIP builds, they can all use their own version and no problem.

If they are system processes, really we possibly need to think about where the MKUUID is held. One system location for it is fine so long as the system is only servicing one client.

If they were legacy applications, each would have its own .ini file and the UUID could sit in that, but a lot are scripted python stuff that doesn't obey the old rules.

Do we need a key pair lookup database (can just be a text file, nothing difficult) which can hold one or more MKUUIDs, allowing the client process to specify which one should be used?

dkhughes commented 5 years ago

@ArcEye

However if /etc/linuxcnc/machinekit.ini exists at package install, the install will fail because it should have been removed and clashes with the file in the package.

Hmm. Well, if the user and our package have updated conf files, dpkg should just ask which to keep right? Problem is, if the user didn't update their conf file, the new one will be copied in silently that has a path variable and we change their UUID. That doesn't really handle the do-no-harm I had in mind.

The default path should always be /etc/linuxcnc/mkuuid If the user wants to use a rotation of keys

and

If they were RIP builds, they can all use their own version and no problem

I wasn't thinking of a rotation of keys for a single instance, but multiple RIP builds on one host that you want to identify uniquely. I was thinking a hardcoded path and symlink to /etc/linuxcnc/mkuuid was just less flexible in that case.

If they are system processes,

Ha, that is tricky. I hadn't thought about the case of doing this after install via package. I was thinking about using multiple RIP with unique UUID to test the trajectory/joint-axis work on multiple machine configs without having to pick the config everytime at the remote gui. I just let the GUI remember each UUID, and it properly pulls the saved settings (work offsets, etc.,). That stuff is transient on my hardware since the zynq image I'm using is based on a RAMdisk to protect the SD card lifetime.

each would have its own .ini file and the UUID could sit in that

This would cause the same problem we're seeing now, where multiple instances on different hardware of the same machine config will publish the same UUID, right?

@cerna

the moment the last service on an instance identified by GUID dies, the instance dies and I care about the GUID no more

That seems fine, but when I use systemd to run mklauncher at boot, the mklauncher instance runs for the entire up-time of that remote piece of hardware - the two launcher services will never disappear. We make sure were talking to the same instance of that service on the hardware by using UUID + service-UUID. The combination tells us the transient info you are talking about. For caching GUI settings, that MKUUID is extremely useful across reboots, even in a DHCP environment where the IP is changed for some reason out of our control. Since MKUUID is in a static config file, I think it should be acceptable to treat it as a key for remembering a specific instance of machinekit.

Edit: Sorry, couldn't let that typo go...their not they're.

ArcEye commented 5 years ago

Hmm. Well, if the user and our package have updated conf files, dpkg should just ask which to keep right?

Unless there is a scripted install, I think not. By default it will overwrite any file which is solely part of the package being installed or will halt and refuse to overwrite a file if it is part of another package. The installation of the machinekit.ini file is looking more and more as though it ought to be a postinst scripted event anyway and no default file copied.

This would cause the same problem we're seeing now, where multiple instances on different hardware of the same machine config will publish the same UUID, right?

I am not promoting it, I was saying it is probably not practical to try and store per config, because all the parsel-tongue boys take great pride in doing away with conventional .ini config files.

It does however touch upon the nub of the problem.

If you are running a package install, which most people do and you want to run multiple instance processes, where are you going to store the MKUUID for each process you want to run?

As soon as there is more than one, you have problems. One solution is to have the UUIDs in key based lookup, which each config can fetch and export when required. That key could be stored in the config somewhere or each instance could even be allocated the next UUID.

I have written another script which does that, get-mkuuid 0 returns the UUID held under key MKUUID_0 in a simple text file.

Taking that to its logical extent, why would you have a different system for RIPs?

None of this affects me personally, on my machines the computer plus software replaces the machine controller, Fanuc or whatever. I don't even use wifi pendants let alone expect a phone or tablet to control something. Wifi just doesn't work in a metal framed concrete building with VFDs etc. throwing out RFI all over the place.

But if we are going to spend time changing the system, it needs to fit all eventualities, not just whatever problems were thrown up with the particular situation you had, or we will just be re-visiting it when another scenario arises.

dkhughes commented 5 years ago

it ought to be a postinst scripted event anyway and no default file copied

I agree, postinst script will let us handle this much more elegantly I think.

Wifi just doesn't work in a metal framed concrete building

Agree, also. But, we use a shielded ethernet network and it works really well, even in the presence of plasma cutting.

on my machines the computer plus software replaces the machine controller, Fanuc or whatever.

The architecture we're using is a little more distributed and I think is more common with those of us using an arm based system. Most arms I've used are just too weak running the graphical interfaces. This allows the one-to-many case I described before.

it needs to fit all eventualities

Agree completely. So, if we use the key/value pair method, how do we decide which key to use? Are you preferring a command line option passed in, like:

> mklauncher -uuid VALUE
ArcEye commented 5 years ago

Agree completely. So, if we use the key/value pair method, how do we decide which key to use? Are you preferring a command line option passed in, like: > mklauncher -uuid VALUE

What I have been playing with is a script to parse the MKUUID from the file based upon the key

I don't use mklauncher so am not familiar with its args but if that line is valid, you could use mklaucher -uuid $(get_mkuuid 2) and $(get_mkuuid 2) would resolve to whatever was held at MKUUID_2 in the file.

This would only require the machine instance to know what number it is, or just how many other instances there are and take the next number, if ID does not have to be persistent per config.

Other scripts could generate or remove keys. They can also make certain there are no duplications, by blacklisting the original default UUID and checking for duplications within the file before writing. The details can be thought through, if something like it is adopted.

It is just my take on having a common method that fits all. Most people would just have one entry in it and be none the wiser as to the mechanics.

cerna commented 5 years ago

Huh, I cannot think of a production scenario in which there is a need for two or more GUID and/or configurations on one installation of Machinekit/one OS, outside of testing and development. Could you please provide one?

(It sounds like a great scenario for if (when) the idea of new configuration stack will be implemented. The GUID which now sits in the machinekit.ini file could sit on the root of the tree which would contain information of current mechinekit.ini, configuration inis and probably some graph representation object which would store how HAL is connected. This would also mean that one system could "instantiate" Machinekit Instances and run them concurrently. Not that I see the reason or need for it. But I don't claim omnipotence.)

So far, wouldn't some wrapper service around current mklauncher started at system start be a better solution for you?

Maybe I took it from a wrong end and maybe I should have implemented pull which would do what link local do, i.e. on the start of service there is ping or query send from new service with proposed name or address and if there is no reply, then the service advertise this name or address as it's own. In the case of reply, it chooses some other name/address. So applying it to Machinekit, the only complaint would be in the case that there already is MKI running with default GUID, otherwise, no error would be raised. So the set of people affected would be a lot smaller.

Since MKUUID is in a static config file, I think it should be acceptable to treat it as a key for remembering a specific instance of machinekit.

Yes, that sounds logical. And I never had anything against it and was actually thinking the same way. It just wasn't that important to me at the moment.

ArcEye commented 5 years ago

Huh, I cannot think of a production scenario in which there is a need for two or more GUID and/or configurations on one installation of Machinekit/one OS, outside of testing and development. Could you please provide one?

No, I don't even use the default one for anything, just trying to cater for eventualities that I'm told exist.

The easiest thing would be to actually use the UUID generated in RIP builds and generate a new one post install for package installs.

Anyone who actually uses a specific UUID for something, will be familiar with the fact it gets overwritten by a new install and will have saved it and will re-instate the UUID they want.

dkhughes commented 5 years ago

The easiest thing would be to actually use the UUID generated in RIP builds and generate a new one post install for package installs.

I think that would handle many cases instead of the static one generated during package build presently. Do we just update the machinekit.ini from a bash script like discussed earlier and invoke that post-install?

Anyone who actually uses a specific UUID for something, will be familiar with the fact it gets overwritten by a new install and will have saved it and will re-instate the UUID they want.

Possibly, but that process can be a lot easier if we add a mechanism to optionally specify a UUID at startup. I like the idea of an additional override. If you don't use it, then nothing is affected. For people like me, we specify something like:

machinekit -uuid=UUID_VALUE /path/to/config

That allows for all of the scenarios we've discussed since we can substitute a random transient id ($(cat /proc/.../uuid), or any other static UUID desired. Nice and flexible.

I added the option to mklauncher already in a local branch. I was going to look at trickling it through to the rest of the required modules today at some point.

Usage would be:

mklauncher -uuid=UUID_VALUE [other options]

Then we can get this issue off the list ;).

ArcEye commented 5 years ago

I'll have a look tomorrow at getting it going from the build and package.

Will leave you to look at the mklauncher side

ArcEye commented 5 years ago

First step, ensure all RIP builds have a unique MKUUID https://github.com/machinekit/machinekit/pull/1447

dkhughes commented 5 years ago

I've been digging trying to figure out how to integrate UUID overriding. Issue is, the machinekit.ini file is parsed separately by almost every module involved in publishing a service. Most can have a simple --uuid option added and invoked from the command line, chained through python/bash calling files already in place. haltalk supports specification through the -R argument, but I'm unclear on the usage intention since that component is specified in a configuration file.

Are command line arguments for machinekit (linuxcnc) forwarded to the display program selected, and the HAL components init string? Is there a mechanism for this already in place that I can't seem to find?

As an alternative, I see that displays support custom variables, but it feels dirty to create a temporary env.ini file for mkwrapper to check for to parse the uuid in the event of an override. That also doesn't address setting the UUID to match in a HAL file specified component. Perhaps adding a UUID value to the ini structure would be a way to unify all these different paths.

I think the issues I'm bumping against can be circumvented with a python based configuration, but supporting legacy HAL/INI files would be nice.

ArcEye commented 5 years ago

The changes for RIPs and packages are in and test building. I made them fairly neutral but adaptable if you have special needs.

RIP builds will leave machinekit.ini alone if it contains a valid MKUUID

Package builds likewise, but if an /etc/linuxcnc/mkuuid file exists with a valid UUID, that trumps all others, allowing the user to maintain a MKUUID across installs. This even after a purge uninstall, which should refuse to delete mkuuid because it was not part of the machinekit package, but created by the postinst script.

Are command line arguments for machinekit (linuxcnc) forwarded to the display program selected, and the HAL components init string?

AFAIK the linuxcnc script, sets and exports the $MACHINEKIT_INI var and it is then up to other elements to fetch that file if it is important to them, parse it and extract the value with getvar() or similar.

cerna commented 5 years ago

@ArcEye I have just tried it and it works well. (Cannot say that I completely 100% agree with this solution and I still don't really see the need for an extra file. But it works and that's the most important thing.)

So, good work and thank you.

ArcEye commented 5 years ago

Thanks for testing and confirming @cerna

I am not convinced myself about the need for the mkuuid file, but as a couple of people seemed to think that persistence might be important as well as uniqueness, that provides the basis for it.

The arcane behaviour of dpkg still catches me out. Just when I thought it was sorted, dpkg was asking me if I wanted to replace my own copy of rtapi.ini with the one in the package. Since /etc/linuxcnc/rtapi.ini didn't exist, it was peculiar.

I figured out that dpkg / apt treats anything in /etc as a configuration file and thus only removes if you do a apt purge machinekit for instance. If you did not purge the package, it assumes the existence of the config files, without actually checking that they do exist.

Another wrinkle to try and store away somewhere so it doesn't catch me next time :)

dkhughes commented 5 years ago

Thanks for the changes @ArcEye! This is working well for me also. I think you've buttoned this issue up.

Given the spread of dependencies on machinekit.ini, updating the existing MKUUID value in this file at install is the cleanest approach for customizing ini values across the different modules, and this gave me a nice example of how to add in a format string for normalizing service announcements.

cerna commented 5 years ago

Maybe just a detail:

On Machinekit forum user Schooner (I guess that's @ArcEye here) wrote a quick note. I think that note should also be in documentation and should be linked from how-to build from source.

ArcEye commented 5 years ago

Yes well prompted. I will cut and paste the text into a document, in case the post disappears for whatever reason

https://github.com/machinekit/machinekit-docs/pull/299 relates