Theme: Interactive Terminal

[JedMeister]

Hi,

Firstly great work on this theme! Secondly apologies if this has already been answered somewhere (I looked but couldn't find anything).

---

We have been using Shellinabox to provide a web based shell, but now with the implementation of your Authentic theme to Webmin, we have a shell there too, so am considering dropping Shellinabox altogether.

However, I notice that the Webmin shell is not picking up the default colorization that I currently get via both an SSH session, or via Shellinabox (see screens below). Also the text is quite small and not that easy on the eye (at least mine anyway).

Is there any way I can adjust that. Primarily I would like to enable colorization of the shell, but it would also be nice to increase the font size a little.

SSH session

Shellinabox

Webmin Authentic shell

[JedMeister]

FWIW it looks like the TERM variable isn't being set. I tried manually setting it within a session, but even if I export it, it doesn't seem to stick?!?

[gnadelwartz]

imho this is not a real shell, each command is executed in seperatly.

[JedMeister]

@gnadelwartz ok thanks for that. I have zero perl experience, but I did have a dig in the shell user interface index.cgi file but couldn't really see what is going on there...

Regardless, I'd still imagine that it must be possible to set/update environment variables?! Even if the renderer didn't know what to do with them (so we'd likely get a heap of escape codes instead of colors).

This isn't a showstopper for me, but it would be nice if I could adjust the font and would be super great if it could have colorized output!

[JedMeister]

Ok, now I'm digging a bit deeper, I'm not sure whether this "pseudo-terminal" shell will be good enough... As it doesn't support interactive commands, I'm looking at other ways to achieve my ends.

I see that there is a shellinabox module (noted at the bottom of this doc page and here is it's directory in the source). I wonder if there is a way that I could leverage that to provide the Authentic theme shell?

I really like everything about the Webmin authentic theme shell, expect the fact that it's not a proper shell. I understand that Webmin is cross platform, but I would imagine that it's most commonly used with Linux (users used to a GUI, managing their Linux servers) so having a "proper" shell would be a huge advantage IMO...

[JedMeister]

Oh, just noticed that the shellinabox module appears to be abandoned (last commit 2012!). There is an ajaxterm module too, but that is also more-or-less abandoned (and the upstream project definitely appears abandoned)...

That sucks! I may see if I can update the shellinabox one easily (but I doubt it). Otherwise, I think I might just stick with Shellinabox for now so as to have a proper shell...

[JedMeister]

I had a quick look at the code and unfortunately, I don't see what I would need to do to import an updated version of Shellinabox into Webmin, but it appears that it should be possible. And if that can be done then, I'm pretty sure that we could then replace the current pseudo-shell accessible within Authentic theme with Shellinabox.

But unfortunately, with my lack of perl knowledge and a big todo list, I'm going to need to put this aside for now...

[gnadelwartz]

@swelljoe @rostovtsev I didn't find the issue, where we discussed replacing current terminal with a modern and real interactive solution, but we should think about a new try after realease.

[iliajie]

Jeremy, hi.

You can easily increase the font of the whole Terminal just by adding to theme's extensions, this CSS code:

.-shell-port- {
    zoom: 1.2
}

Note: After saving changes, for now, you would need to refresh the browser's page to get the effect.

What's for interactive shell. We're all looking forward to have it.

I will think more on how it could be done. Fedora Cockpit has it for instance. Shellinabox has unofficial fork. The problem is to make it work on all Linux distros. We could make a build and create AppImage to make sure that it just runs without hassle. After all, I would need to make a port to the UI.

I will mark this issue as an enhancement and will look forward for enrolling it. It will be for sure after the new mail is released, into more or the less stable condition, which estimated to happen around summer 2018.

I absolutely agree, that having proper interactive console is a huge win. I'm making it a second priority after the mail.

[iliajie]

I will look into Gate One as possible choice.

[gnadelwartz]

this would be ny initail suggestion also, phyton and javascript should be portable

[iliajie]

I managed to install Wetty in no time.

Requires only NodeJS.

[gnadelwartz]

nice, but this adds a new dependency :-/ nodejs may not installed everywhere... at least on embedded systems ...

May be we have to keep actual implementation as fallback.

[iliajie]

We could try to make AppImage, just export all of the dependencies into single, self-contenting package.

We could surely keep current implementation as a fall-back.

[jcameron]

Yeah, I'd rather not depend on node.js in Webmin. Can the server-side part of this be re-implemented in Perl?

[iliajie]

Reimplemented? Technically but not practically. It's very time consuming. Besides, there is implementation in Perl (Shellinabox) already.

I haven't looked very closely yet to make a judgement.

Why we just can't ship AppImage - single, executive file that would contain all dependencies in one. I'm not saying it's the best way, but it looks for now as the fastest solution.

[gnadelwartz]

I would also prefer a perl or phyton inplementation (as we currently has).

Is you appimage sulotion is portable or does it depend on OS and CPU architecture?

[swelljoe]

Requires only an entire new runtime, eh? ;-)

Requiring Node is too much, I think, as it is not commonly installed and is very, very, large. But, then again, the alternatives aren't great either. Python can be expected to be available everywhere, already, but I dunno if there's a WebSockets based back end available that'd work with one of these modern JS terminal implementations. There's one in Go called gotty, but if we shipped a single binary it'd only work on Linux systems (the server-side, obviously the client-side is JS and would work anywhere).

I dunno. We pretty much have to spin up a new back-end webserver, no matter what, as miniserv.pl doesn't support websockets, yet. Python is more readily available than Node (for now), but I don't know of good options for that. I haven't looked at Shell In a Box, I think it predates websockets, though, so I doubt it's the modern approach we'd want.

There is a WebSockets implementation for Perl in Mojo (as well as several compiled ones, but I think Mojo is good as it has fewer dependencies than most), but someone would need to implement the shell backend (I don't know what that entails...we'd basically be porting the backend in Wetty or Gotty to Perl). I'll have a look at those backends and see how much code we're talking about...I think the heavy lifting is on the front, so maybe it wouldn't be so awful to port it to Perl.

[swelljoe]

Maybe this: https://github.com/Prajithp/mojo-terminal

[gnadelwartz]

As noted above Gate One at least state:

• Gate One works with Python 2.6+, Python 3, and even pypy!
• The daemon that acts as the web server for Gate One is small and light enough to be included in embedded devices.

https://github.com/liftoff/GateOne/

[gnadelwartz]

Maybe this: https://github.com/Prajithp/mojo-terminal

wow. seems really small (execpt of require mjojo)

BTW: nice to see all of you active in the same time frame :-)

[JedMeister]

Great work guys, pumped about all the activity... :smile:

Gate One looks good.

I agree depending on Nodejs is not desirable!

Out of curiosity, why is Shellinabox out of contention? IMO it works quite well. Although it is a little flaky on mobile devices...

[swelljoe]

mojo-terminal looks very good, if it is in a functional state. It's very light, only has a few dependencies (still checking to see if they are pure-Perl or if there's some compiled bits), and it uses the same frontend as all of the others we're talking about (xterm.js), so it'd be something we could sort of pin to whatever version is current and just perform minor updates based on what upstream does. I think this might be a winner.

A couple of XS dependencies (IO::Tty, though we already use IO::Pty, which is a similar size/complexity dependency, so this is probably not a dealbreaker). I'm looking at the dependencies now to see which ones are already in CentOS/Debian repos, and figuring out if we'd need to package any extra modules. Mojo is not in the CentOS core repositories, but I think it is in EPEL. And, it's pure-Perl and only a couple thousand lines of code, so we could even bundle the damned thing as part of Webmin.

Mojolicious is really an amazing bit of code. It does so much with so little code. I'm in awe of it every time I look at stuff implemented with it.

[gnadelwartz]

Out of curiosity, why is Shellinabox out of contention? IMO it works quite well. Although it is a little flaky on mobile devices..

Not out of scope but if we implement a new solution it has to fit some requirements:

modern(e.g. websockets) , futureproof, portable (perl, phyton)...

BTW: cpan Mojolicious works as a charm on my embedded NAS

[swelljoe]

I don't know that Shellinabox is out of contention...I just know it's really quite old, and I doubt it is a modern implementation. With Websockets, it is now possible to have a very efficient communication between the browser and the server; but it's a relatively recent thing. I think Shellinabox pre-dates it. If it does, the implementation would be poll-based, and also very large compared to these WebSockets-based implementations. The smaller it is the better, because we can probably maintain a few hundred lines of code if it goes abandoned in the future, but we can't realistically maintain or understand or audit thousands of lines of code (and security is a big issue with anything we ship in Webmin).

So...if we can find an implementation of this that is:

1. Small. A few hundred lines of code, hopefully.
2. Doesn't require a new runtime: Limited to Perl or Python, probably.
3. Works with a well-maintained frontend. I think this is just xterm.js, as I don't know of any other frontends that are widely used.
4. Has minimal dependencies, preferably no or few binary packages. I suspect any Perl thing will require at least an XS module or two for the IO, but that's probably OK.

[JedMeister]

Sounds great. Thanks for the explanation, that makes tons of sense. And yes, Shellinabox is old. It died but has been (somewhat) resurrected, but AFAIK no major changes to the codebase (just some updates and bugfixes).

It looks like you guys are all over it.

In the meantime, if I come across anything that fits the bill, I will certainly let you know.

[swelljoe]

So, for dependencies:

• CentOS core has perl-IO-Tty
• We already use perl-IO-Pty (though I dunno where we get it from, as it's not in core, maybe it's in EPEL or we've got a build of it somewhere in our repos).
• Mojo isn't in either core or EPEL, but the bits we need are pure-Perl, as far as I know, and only a couple thousand lines of code, so we could bundle the damned thing with Webmin (which is several hundred thousand lines of code, so not a big deal).

I think that's realistic. So, I'll need to try it out to make sure it's actually in a usable state...sometimes projects get kinda half finished and the maintainer gets bored and moves on before it's solid.

There will need to be code to spin up the sockets server on demand. I don't know how authentication is handled in the AJAXTerm module (which I think is the closest comparable Webmin module), maybe you actually have to login again...I don't remember. But, we'll have to make sure we get that bit right. Ideally, it wouldn't require logging in again, so I guess we'd need to set a JSON Web Token to authenticate the current session, maybe.

@qooob what are you doing about authentication when you use wetty in this context? Is it requiring a new login or are you setting a token?

[JedMeister]

Let me know if anything needs testing and/or you want any further feedback.

[gnadelwartz]

Mojo isn't in either core or EPEL, but the bits we need are pure-Perl

as long CPAN is working it can simply installed

perl-IO-Tty ... perl-IO-Pty

I see also no problem with this, may be we can unify code to one of them ...

[gnadelwartz]

Great work guys, pumped about all the activity...

as you can imagine, we are not really statisfied by current terminal :-)

BTW: what about a picture of a tasmanian devil ;-)

[swelljoe]

I don't like to rely on CPAN for Perl modules in anything we ship in Webmin core, or in Virtualmin. Jamie is more relaxed about this than I am, but if we can't include it as a dependency in the Webmin package, I wouldn't want to make it part of core Webmin (this means it needs to be available in the core OS repos, which IO::Tty is, and I see it includes IO-Pty).

So, Mojo is the only unresolved dep, I think. One other XS module needed is Encode, but it's also in the core CentOS repos (and I just assume that anything CentOS has, Debian/Ubuntu will have because they have a lot more Perl packages). So...the new terminal module probably just needs to embed Mojo in a directory and add it to the library path when it spins up the server.

We may still want to make this a non-core module that gets installed as part of Virtualmin and Cloudmin. If there are other XS (binary) dependencies hiding in here that I haven't found yet, and they aren't available in the core OS repos so we can depend on them in the Webmin package, I'd want to see this as an external module. But, it looks like that won't be necessary...we can probably ship all the non-core bits we need in the Webmin package.

And, yeah, the current popup terminal in Authentic is awesome, but it's not really a terminal...it's a command shell that looks really cool. It's never been intended to replace an ssh session, but the tech exists to replace ssh sessions (or put them on the web), we just don't have it in Webmin yet.

[JedMeister]

And, yeah, the current popup terminal in Authentic is awesome, but it's not really a terminal...it's a command shell that looks really cool. It's never been intended to replace an ssh session, but the tech exists to replace ssh sessions (or put them on the web), we just don't have it in Webmin yet.

Yeah it tricked me initially! It wasn't until I followed the hint from @gnadelwartz and I dug in a little that I realised what was going on. Even as it is, It's certainly handy (and looks nice) but I was looking to replace our current (standalone) Shellinabox implementation. I don't need a perfect SSH replacement, but I do ideally want a Shellinabox replacement! :smile:

Despite Shellinabox being old (and a little bit clunky) it works quite well. But I'd much rather consolidate all the additional remote access stuff within Webmin (so no need for an additional service and port open). If you guys find and can implement something even better than Shellinabox, I'd be super keen!

Oh, and as requested by @gnadelwartz :smile: :

[iliajie]

I tend to use mojo-terminal as well. I will have to take a closer look first though. I tend to use Perl as back-end as well for better overall compatibility.

@qooob what are you doing about authentication when you use wetty in this context? Is it requiring a new login or are you setting a token?

Wetty runs /bin/login, so no, plain authentication with login/password. It supports other methods, supposedly even tokens and other but I dind't try it.

[swelljoe]

I started tinkering with mojo-terminal, and it is functional, however, it has no authentication...it doesn't even ask for a login, it just runs as the user that started the service.

This means it needs an authentication layer implemented at the Mojo level, and we need to be able to set a session cookie in a Mojolicious compatible form within Webmin for each user with an active terminal.

We also need to be able to spin it up when the user asks for it and kill it when the user logs out (this will mean it is slower than the current popup terminal on first launch, but not by much...on my laptop, mojo-terminal is able to answer requests in less than a second). But, each process requires about 200MB/37MB (virt/res), so we can't just spin it up on every login, as any system with more than a few active users would end up pretty big pretty fast. I think the spin up/down code mostly already exists in the AJAXTerm module, so I'll look at that to see if I can steal it. But, authentication will be harder, I think. Webmin's session cookies aren't the same as Mojo's (though maybe I can convince Jamie to implement JWT auth in Webmin, which would be re-usable against Mojo). I'm a bit iffy on how authentication across ports can work.

Mojo can also listen on a UNIX socket (a local file), but I don't think Webmin can proxy WebSockets so that probably won't work, anyway...though it'd solve the authentication problem, if it could, since we could create a file for each user owned by the user and Webmin could just talk to it as the logged in user, (or root), and even local users wouldn't be able to abuse that.

[iliajie]

Sounds good. I'm a little baffled, is there really that much resources needed?

It would be great to have it replacing old AJAXTerm in the end.

[swelljoe]

I mean, each process is 40MB resident (200MB virt). That's not "much" by many standards...but, on small machines that are already packed to the gills (a 1GB machine running a Virtualmin default setup, including clamd, there is no room to spare), it could put the server into swap hell with just a few users spinning of concurrent shells. That's more than spawning Command Shells on demand, by a lot.

But, yes, it's probably smaller than AJAXTerm. But, we don't put AJAXTerm into a spot where hotkey can pop it up for any logged in user.

It's just a matter of managing expectations, I think. This is going to spin up a process that's as big as, or potentially bigger than, Webmin itself, for every user that spins up a terminal. That isn't intuitive; who would expect a shell to be as big as the GUI? Certainly much of it will be shareable, as it's all Perl (this is another good reason to use something in Perl...we know the shared system libraries are gonna be mostly the same for Webmin and the other Perl processes).

Anyway, it's fine. It's fast enough, I think, and small enough. I was just sort of notating what I'd found in looking at it, so we know what we're dealing with. I'll fork it and add authentication tomorrow, and then from there, maybe get Jamie to sort out the part where Webmin auto-authenticates (I don't understand Webmin's session handling or how it makes authenticated calls to things like AWS, which is kinda what'll be happening here).

[gnadelwartz]

@swelljoe any progress on this topic?

[datavectors]

Since there have been no follow up posts ...

Regarding colourisation of output from commands ... I have a Custom Command which looks like this ..

/etc/ansible/inventory/gce.py --list | python -m json.tool | pygmentize -l json | aha

This prints out a list of inventory assets in colourised syntax (json).

The custom command must be set to HTML output for this to work. Also in Ubuntu aha must be installed and python pygmentize module.

gce.py is a standard script used in Google Cloud API.

...

Returning to exploring a standard shell module, I have looked at wetty (which requires node.js) but I am now focussing on building butterfly into a custom module. My aim is to build a module to run Ansible scripts from within Webmin framework. i.e. using webmin as the controller for running Ansible scripts on both the local server and satellite servers.

butterfly with SSH

[michacassola]

I use webmin mostly for the file manager to edit files but being able to use it also for the command line would be a really nice thing. Are there any final conclusions of what software to use and when you can implement it?

@swelljoe said in a referenced issue that some heavy lifting is involved for both back and frontend.

[swelljoe]

There won't really be a "final conclusion" until we're able to put the time in to make it work. And, it is still a relatively large amount of work remaining.

I don't necessarily agree with closing the ticket until it's done, but Ilia likes to keep a clean issue tracker (which is also valuable...and something we don't do well in some of our others). His thinking is that it's on our todo list, so it'll get done and there isn't much else to discuss about it...but, my thinking is it doesn't exist yet, so probably should still have an open ticket.

But, this is Ilia's tracker, so he runs it his way. ;-)

Anyway, it's coming...but, could take a while.

[michacassola]

@rostovtsev and @swelljoe

Any news on this?

[probonopd]

Let me know if you need help on producing an AppImage. https://docs.appimage.org/

[swelljoe]

Nothing interesting. I have been tinkering with Mojo Terminal, and kinda have a vague notion of how we can implement authentication safely, but it's still theoretical, and my knowledge of Mojolicious is pretty weak. I'd like to be able to ship it in time for Virtualmin 7, but I'm not super confident I'll be able to sort it out by then with everything else I've got going on.

[michacassola]

Isn't Virtualmin 7 the second next version only? Meaning years?

Any good news @swelljoe ? Also: Thank you for your last update!

[iliajie]

@michacassola For your information, latest theme release let's you run edit /file/name/to/edit, to bring up the internal file editor.

[michacassola]

@rostovtsev Thank you for that info. I do not know though if I need that as I usually use sudo nano /path/to/file when I ssh into the machine. And I love the file manager in webmin to do it the UI way! That is my favourite already complete feature (although I would love it if it is faster).

I would love to run interactive commands. Some apps don't let you put all the parameters into the first command like 'app do this -y'. As a webhoster webmin has made my life easier already thank you for that. I would love interactive SSH though. And I will also add some custom functions in the future if a future client might require it. But for now I would love to just use the online shell and not have to use two programs for server management. But of course it is just a nice to have feature.

[swelljoe]

No, we're on Virtualmin 6 currently. 7 is coming within the next few weeks, probably, depending on how much time I have to devote to it.

[michacassola]

Hey @swelljoe, I was very happy about the new monitor on the dashboard, how about the interactive Terminal? Any news? And thank you for all the work already done! :)

[swelljoe]

Thank @rostovtsev for the new realtime monitoring in the dashboard (and it is very cool!), as it's all his doing.

Nothing new. I had some personal stuff that kept me away from doing much work on Virtualmin, and took a now job at the beginning of 2019 that also took (and continues to take) a lot of my time. With what time I do have (weekends and holidays, mostly), I've been focused on catching up on OS support (Debian 10 and CentOS 8), and fixing some major problems with the website (replaced forums during Christmas holiday, and will hopefully have commerce and issue tracker replaced over the next couple of weeks).

[iliajie]

Thank @rostovtsev for the new realtime monitoring in the dashboard (and it is very cool!), as it's all his doing.

Thank you for your kind words, Joe! :smile_cat:

.. and will hopefully have commerce and issue tracker replaced over the next couple of weeks.

Looking forward to it!

[michacassola]

Hey @swelljoe, how is your job going? Hope you guys are all fine! :)