augeas lenses: test in- and export

[Namoshek]

As the snippet sharing service attempts to export a snippet in all formats when you visit the details page, it makes sense to disable lenses that do not work properly. To find out which lenses that are, I'd suggest trying one or two snippets per lens. Everyone is welcomed to add test results to the list!

A list of augeas lenses with tests:

lens	import	export	test snippet
access.lns	ok	ok	cat /etc/security/access.conf
activemq_conf.lns	ok	ok	random sample
activemq_xml.lns	ok	X	current default config
aliases.lns	ok	ok	cat /etc/aliases
anacron.lns	ok	ok	cat /etc/anacrontab
approx.lns	ok	ok	random sample
apt_update_manager.lns	ok	ok	random sample
aptcacherngsecurity.lns	ok	ok	random sample

aptconf.lns aptpreferences.lns aptsources.lns authorized_keys.lns automaster.lns automounter.lns avahi.lns backuppchosts.lns bbhosts.lns bootconf.lns build.lns cachefilesd.lns carbon.lns cgconfig.lns cgrules.lns channels.lns cobblermodules.lns cobblersettings.lns collectd.lns cron.lns crypttab.lns cups.lns cyrus_imapd.lns darkice.lns debctrl.lns desktop.lns device_map.lns dhclient.lns dhcpd.lns dnsmasq.lns dovecot.lns dpkg.lns dput.lns erlang.lns ethers.lns exports.lns fai_diskconfig.lns fonts.lns fstab.lns fuse.lns gdm.lns group.lns grub.lns gtkbookmarks.lns host_conf.lns hostname.lns hosts.lns hosts_access.lns htpasswd.lns httpd.lns | ok | X | cat /etc/apache2/sites-available/www.libelektra.org-le-ssl.conf inetd.lns inifile.lns inittab.lns inputrc.lns interfaces.lns iproute2.lns iptables.lns jaas.lns jettyrealm.lns jmxaccess.lns jmxpassword.lns json.lns kdump.lns keepalived.lns koji.lns krb5.lns ldif.lns ldso.lns lightdm.lns limits.lns login_defs.lns logrotate.lns logwatch.lns lokkit.lns lvm.lns mcollective.lns mdadm_conf.lns memcached.lns mke2fs.lns modprobe.lns modules.lns modules_conf.lns mongodbserver.lns monit.lns multipath.lns mysql.lns nagioscfg.lns nagiosobjects.lns netmasks.lns networkmanager.lns networks.lns nginx.lns nrpe.lns nsswitch.lns ntp.lns ntpd.lns odbc.lns openshift_config.lns openshift_http.lns openshift_quickstarts.lns openvpn.lns pam.lns pamconf.lns passwd.lns pbuilder.lns pg_hba.lns php.lns phpvars.lns postfix_access.lns postfix_main.lns postfix_master.lns postfix_transport.lns postfix_virtual.lns postgresql.lns properties.lns protocols.lns puppet.lns puppet_auth.lns puppetfileserver.lns pythonpaste.lns qpid.lns quote.lns rabbitmq.lns redis.lns reprepro_uploaders.lns resolv.lns rsyncd.lns rsyslog.lns rx.lns samba.lns schroot.lns securetty.lns sep.lns services.lns shells.lns shellvars.lns shellvars_list.lns simplelines.lns simplevars.lns sip_conf.lns slapd.lns smbusers.lns solaris_system.lns soma.lns spacevars.lns splunk.lns squid.lns ssh.lns sshd.lns sssd.lns stunnel.lns subversion.lns sudoers.lns sysconfig.lns sysconfig_route.lns sysctl.lns syslog.lns systemd.lns tests thttpd.lns up2date.lns util.lns vfstab.lns vmware_config.lns vsftpd.lns webmin.lns wine.lns | X | X | uploaded as wine-cfg xendconfsxp.lns xinetd.lns xml.lns xorg.lns xymon.lns xymon_alerting.lns yum.lns ...

[markus2330]

@ElektraInitiative/elektradevelopers Can you please help by uploading some (valid and even better: helpful) snippets and choosing one of the formats above? Thank you!

[Namoshek]

Maybe it would also make sense to note the format used by the app, because we don't need 20 ini plugins and we could disable some of them..?

[markus2330]

ini is something very special: there are hundrets of incompatible dialects out there.

Btw. wine does not work. (Not even parsing, not even a unmodified generated file). It even says in the lens: "This is pretty quick and dirty").

Btw. can we change the timeout of the backend? For larger files it always runs into timeouts. Or should we give some length restrictions of the snippets? For wine-cfg even the viewing of the snippet runs into timeout?

Disabling plugins would improve the situation, too...

[Namoshek]

Yes, I think we should disable plugins, not only because of timeouts, also because performance is already bad for one-liners. I understand your issue with ini dialects though.

I'm trying most imports/exports on my debian VM by the way (mostly because I don't know if I may share a snippet I find randomly on the web); it is so much faster than the website. I think the main issue we have is the storage. Is there a more performant way to convert a snippet than writing to a temporary file? (if it was never written to the disk, it might be a lot faster, don't really know)

Btw. please re-open the edit mode for the post shortly before you write something, two of my tests disappeared. :crying_cat_face:

[manuelm]

Is there a more performant way to convert a snippet than writing to a temporary file?

memmapped file or tmpfs for the easy way

[markus2330]

Yes, I think we should disable plugins, not only because of timeouts, also because performance is already bad for one-liners. I understand your issue with ini dialects though.

Ok, then there is no way around it: let us disable most of augeas. (Except of some hand-picked ones like hosts and the ones you tested)

I'm trying most imports/exports on my debian VM by the way (mostly because I don't know if I may share a snippet I find randomly on the web); it is so much faster than the website. I think the main issue we have is the storage. Is there a more performant way to convert a snippet than writing to a temporary file? (if it was never written to the disk, it might be a lot faster, don't really know)

Where does writing to a temporary files happen? As @manuelm said: We could write to tmpfs. But the website might not benefit a lot because memory is quite limited (and swap is not fast either).

But without benchmarking everything it is speculative guessing.

[Namoshek]

Where does writing to a temporary files happen?

The service_convert.cpp gives a temporary file to the plugins to do kdbSet/kdbGet. If an in-memory file would use swap, the advantage would be zero, but otherwise I think an in-memory file could improve speed. But as I don't know how slow the augeas lenses really are, it's only speculation. Could also be that augeas is the actual reason and not the files.

[markus2330]

Augeas lenses are really slow, but for small files it should not matter. There is also a lot of other potential for optimization (e.g. not reloading all plugins all the time), but these are certainly not things to do quickly before release.

[Namoshek]

I agree, but iterating 182 augeas lenses + the other plugins is just a bit much. What lenses do you think we should keep? My list would be for the moment: access, aliases, anacron, cron, desktop, fstab, hosts, iptables, json, lvm, mongodbserver, mysql, nagioscfg, networks, nginx, ntp, ntpd, odbc, php, phpvars, postgresql, puppet, samba, schroot, services, ssh, sshd, subversion, xml (without testing all of them)

Alternatively, but I also think we shouldn't make this change now shortly before release, we could omit plugin variants (maybe except the one that was used for upload) from the default output and add something like a lazy load (in the front-end) with the help of a dropdown. Or maybe keep the API as it is and add an additional GET flag that enables this behavior, so we keep backwards compatibility.

[markus2330]

Yes, your list looks fine. What about adding some text that people should report an issue if they want additional plugins/lenses? (Say that not all are activated)

output and add something like a lazy load

That sounds very useful in any case. Any reason why we do not have this already?

keep backwards compatibility

not really an issue before the release

Should we tag the release or is it enough to write it as text in the commit message when moving the news?

[Namoshek]

Yes, your lists look fine. What about adding some text that people should report an issue if they want additional plugins/lenses? (Say that not all are activated)

Better give a hint that they can use custom configuration for disabled lenses and where they can find a list of lenses?

That sounds very useful in any case. Any reason why we do not have this already?

Not thought about it. Before the augeas variants we also never had performance issues to be honest. I disabled most variants now (extended the list from above a little bit, I overlooked some lenses that seem useful, e.g. properties). It is a lot better now.

Should we tag the release or is it enough to write it as text in the commit message when moving the news?

Add a tag, it could be useful later.

[markus2330]

Better give a hint that they can use custom configuration for disabled lenses and where they can find a list of lenses?

We might not always have the latest version of augeas, but seems like it is the easiest way. Alternatives like sharing /usr/share/augeas/lenses/dist/ read-only via webserver or have the lenses as repo does not sound so good, does it?

Not thought about it. Before the augeas variants we also never had performance issues to be honest.

Okay ;) If it is an easy fix, it would be great to have it lazy.

Add a tag, it could be useful later.

Which name? restapi1?

[markus2330]

it is the easiest way

Looking a bit more closely at the website: actually 3 of the 4 columns are useless for our users.

[Namoshek]

We might not always have the latest version of augeas, but seems like it is the easiest way. Alternatives like sharing /usr/share/augeas/lenses/dist/ read-only via webserver or have the lenses as repo does not sound so good, does it?

I'm not a fan of it to be honest. If they chose a lens that is not installed, they should just get the info that the format is not supported or something like that.

If it is an easy fix, it would be great to have it lazy.

Doable, but not easy. I'll think about it first, don't want to introduce any issue.

Which name? restapi1?

Better go for website-release or something like that, we will remember this easier than the restapi stuff.

Looking a bit more closely at the website: actually 3 of the 4 columns are useless for our users.

I also think that slug can be avoided (as it is the title in slug form most times). What other columns you don't like?

[markus2330]

Doable, but not easy. I'll think about it first, don't want to introduce any issue.

Then it is clear: lets do it later. We should not do risky things now, we are already overtime ;)

Better go for website-release or something like that, we will remember this easier than the restapi stuff.

So you consider the restapi only released with 0.8.20? We should write so in the release notes.

I also think that slug can be avoided (as it is the title in slug form most times). What other columns you don't like?

I mean the augeas site :grin:

The slug with default value is okay, it was only annoying when you had to type something in there. I sometimes removed redudant info from the slug, so the solution is quite nice after all. Certainly not something that needs a quick fix now.

[Namoshek]

So you consider the restapi only released with 0.8.20? We should write so in the release notes.

No, the restapi is released, don't worry. I just think that a tag, which helps in finding versions, should be more frontend-related as we talked more about this part and therefore will remember longer...

I mean the augeas site :grin:

Well then we could also do ls -1a /usr/share/augeas/lenses/dist >> /srv/libelektra/augeas_lenses.txt and tadaa... :grin:

The slug with default value is okay, it was only annoying when you had to type something in there. I sometimes removed redudant info from the slug, so the solution is quite nice after all. Certainly not something that needs a quick fix now.

And here I was talking about the snippet search table. :grin: (so only a display issue, no real change)

[markus2330]

No, the restapi is released, don't worry. I just think that a tag, which helps in finding versions, should be more frontend-related as we talked more about this part and therefore will remember longer...

Ok, we can also set two tags.

I will move the news then (without tags at the moment)

Well then we could also do ls -1a /usr/share/augeas/lenses/dist >> /srv/libelektra/augeas_lenses.txt and tadaa... :grin:

Ok ;)

I submitted the first news (only link), please upvote:

https://news.ycombinator.com/item?id=13238374

[Namoshek]

I started a build, it should be visible soon!

[markus2330]

Is there any change you want to do in the repo, or can we release as-is?

[markus2330]

Found one more wrong link in the website... I started build again.

I posted the news already at most sites, the tag, however, is still not pushed.

[Namoshek]

I think you can release it as-is. Changes will happen, but before we never get out a release, we should at least start somewhere. With 0.8.20 we can do a second one then if necessary.

[markus2330]

Sounds good! I'll push the tags!

Now we need to keep the noise on the website low (e.g. cancel build jobs) so that it does not start to lag ;)

[markus2330]

Uploading is progressing slowly (or not at all). Are there issues with uploading? For me uploading worked (9 -> 10 snippets). Isn't the time of year about sharing? :yellow_heart:

[Namoshek]

I added some apache configurations. I've chosen some special ones that show different use cases (redirect, scgi, alias, proxy, etc).

[markus2330]

Thanks!

Unfortunately, the news sites did not report our story yet. I also posted it in some mailinglists, but no feedback so far. It is unusual quiet, maybe it is not a good time of the year after all? We will definitely make it a thing again for 0.8.20. In any case it will be useful within the Elektra community with curlget and for posting config files that do not fit nicely in github issues.

Maybe people visited the page and only used the snippet conversion, but did not upload something? We should have a look at the apache logs in the next year to investigate that. I also summarized points we discussed how we could improve the website even further, see #1228 please feel free to add something there.