search

python-distro / distro

A much more elaborate replacement for removed Python's `platform.linux_distribution()` method
https://distro.readthedocs.io/
Apache License 2.0
264 stars 66 forks source link

CloudLinux VM not recognized #240

Open adamjstewart opened 5 years ago

adamjstewart commented 5 years ago

I've encountered a CloudLinux VM that distro is unable to recognize:

$ distro
Name: 
Version: 
Codename: 
$ python -c 'import distro; print(distro.linux_distribution())'
('', '', '')

The problem is that none of the normal search strategies work:

$ which lsb_release
/usr/bin/which: no lsb_release in (...)
$ ls /etc/*release
ls: cannot access /etc/*release: No such file or directory
$ ls /etc/*version
ls: cannot access /etc/*version: No such file or directory
$ uname -a
Linux web.illinois.edu 3.10.0-962.3.2.lve1.5.24.9.el7.x86_64 #1 SMP Wed Feb 13 08:24:50 EST 2019 x86_64 x86_64 x86_64 GNU/Linux
$ cat /proc/version
Linux version 3.10.0-962.3.2.lve1.5.24.9.el7.x86_64 (mockbuild@buildfarm02.cloudlinux.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Wed Feb 13 08:24:50 EST 2019

The only strategy that seems to work is:

$ hostnamectl
   Static hostname: web.illinois.edu
         Icon name: computer-vm
           Chassis: vm
           Boot ID: 5b5b8c522ccd4974808f192aea001491
    Virtualization: kvm
  Operating System: CloudLinux 7.6 (Vladimir Lyakhov)
       CPE OS Name: cpe:/o:cloudlinux:cloudlinux:7.6:GA:server
            Kernel: Linux 3.10.0-962.3.2.lve1.5.24.9.el7.x86_64
      Architecture: x86-64

We may need to add another search strategy to support this type of VM.

HorlogeSkynet commented 2 years ago

Hey @adamjstewart, do you know where the CPE is provisioned in such an environment ? Thanks, bye 👋

adamjstewart commented 2 years ago

What is a CPE?

HorlogeSkynet commented 2 years ago

Common Platform Enumeration, look at hostnamectl output. I'll do some research on my own soon.

adamjstewart commented 2 years ago

Hostnamectl output is attached above. Is there something else you need from me?

HorlogeSkynet commented 2 years ago

So we are looking for OperatingSystemPrettyName and OperatingSystemCPEName D-Bus attributes (see sources).

If we can manage to fetch them without reading from the bus (maybe they are provisioned on the file system too ?), we could then implement a trivial CPE parser and extract information from there.

adamjstewart commented 2 years ago

I recursively grepped the entire OS for those strings but didn't find anything useful. Maybe it would be easier to build a hostnamectl parser?

HorlogeSkynet commented 2 years ago

Thanks for trying this. hostnamectl --json=short could make this straightforward.

Maybe we are missing something, and others got a better idea ? Another way would be to read from D-Bus directly... A Python dependency is required though. See you 👋

adamjstewart commented 2 years ago

This version of hostnamectl doesn't have a --json option.

HorlogeSkynet commented 2 years ago

So re.search around hostnamectl output, or something like dcar if we want to stop calling binaries directly in the future.

NebularNerd commented 3 months ago

Coming to join the conversation here as my issue #368 was basically a duplicate.

Happy to help test any potential solutions on my GoDaddy CloudLinux 8 server. As discussed in my issue hostnamectl is not available in that environment.

HorlogeSkynet commented 3 months ago

Hello again @NebularNerd !

We can start by checking whether the above "D-Bus way" works in your environment too.

bustctl is another systemd binary but if you have it, could you run : busctl get-property org.freedesktop.hostname1 /org/freedesktop/hostname1 org.freedesktop.hostname1 OperatingSystemPrettyName ? If it isn't, could you give a try to dbus-send --system --print-reply --dest=org.freedesktop.hostname1 /org/freedesktop/hostname1 org.freedesktop.DBus.Properties.Get string:org.freedesktop.hostname1 string:OperatingSystemPrettyName ?

Thanks ! Bye :wave:

NebularNerd commented 3 months ago

Hello again @NebularNerd !

We can start by checking whether the above "D-Bus way" works in your environment too.

bustctl is another systemd binary but if you have it, could you run : busctl get-property org.freedesktop.hostname1 /org/freedesktop/hostname1 org.freedesktop.hostname1 OperatingSystemPrettyName ? If it isn't, could you give a try to dbus-send --system --print-reply --dest=org.freedesktop.hostname1 /org/freedesktop/hostname1 org.freedesktop.DBus.Properties.Get string:org.freedesktop.hostname1 string:OperatingSystemPrettyName ?

Thanks ! Bye 👋

Hi @HorlogeSkynet 🙂

HorlogeSkynet commented 3 months ago

What a shame... I wonder whether we should opt for a simpler approach like checking for /usr/sbin/cloudlinux-selector presence, and hope for upstream to properly include an os-release (or any other indicative) file. What do you think ? 🙄

Note : this wouldn't give us the version, nor the "pretty name".

EDIT : if we wanna stick to the D-Bus way (and if we cannot manage to directly retrieve what D-Bus queries), we have to come up with a first Python script to dialog with D-Bus if you don't have any system tool to do so 🤡

NebularNerd commented 3 months ago

/usr/sbin/cloudlinux-selector does exist and gives a python file with the following:

# -*- coding: utf-8 -*-
# cloudlinux-selector Utility to check Cloudlinux license
#
# Copyright © Cloud Linux GmbH & Cloud Linux Software, Inc 2010-2022 All Rights Reserved
#
# Licensed under CLOUD LINUX LICENSE AGREEMENT
# http://cloudlinux.com/docs/LICENSE.TXT

from __future__ import print_function
from __future__ import division
from __future__ import absolute_import
import sys

from clselector.cl_selector import CloudlinuxSelector

def main(argv):
    """
    Main run function
    """
    cll = CloudlinuxSelector()
    return cll.run(argv)

if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

So that could at least says we are on CloudLinux if nothing else. Proper 🤡 OS, so locked down, even pandas does not work quite right under their Python 3.11 environment.

HorlogeSkynet commented 3 months ago

Thanks for this ! I've browsed the Internet, and nothing comes up for such a Python package/module... I'd like to take a look at these sources, just in case they include some information about the environment, or maybe CloudLinux version (?).

Could you check /opt/alt/python3*/lib/python3.*/site-packages/clselector/ and /opt/alt/python3*/lib/python3.*/site-packages/clselect/ packages ? From some random forums, clselector Python packages seems to be installed there.

If Pandas fails to work in a simple Python environment, I fear dealing with D-Bus (is it even installed and running ?) might be complicated as well...

Thanks (and good luck), bye :wave:

NebularNerd commented 3 months ago

It's not that Pandas fails but it behaves very oddly, I have a script that happily talks to my MariaDB SQL database via PD on Windows over a SSH Tunnel, but through GoDaddy CloudLinux on the server where the database is located it starts flagging up messages about not liking the database anymore.

For /opt/alt/python3*/lib/python3.*/site-packages/clselector/ and /opt/alt/python3*/lib/python3.*/site-packages/clselect/:

Not found in:

HorlogeSkynet commented 3 months ago

So I initially planned to directly load this Python module and retrieve/infer its revision and maybe remap it to CloudLinux own version, but :

So our options would be :

  1. Simply check for cloudlinux-selector presence and infer "CloudLinux" as pretty name and normalized identifier (leaving version field blank as we don't have any information about that) ;
  2. Continue our reverse engineering of "clselect(or)" ;
  3. Spend more time on the "D-Bus way" to retrieve and parse CPE ;
  4. (wait for upstream to ship an identifying file in rootfs 🤡).

What do you think ? My 2 cents would be to opt for the quick win 1, even without version.

Bye ! 👋

NebularNerd commented 3 months ago

It certainly looks like it's closed source (or at least not easily obtainable source) I imagine in part they do it to make the server instances secure. Looking at the decompiled code it's wrote in fairly old Python as well.

They have a GitHub but navigating that is not straight forward. Personally, I think reliance on unknown external packages could be an issue later on.

Option 1 provides the simplest route for a quick and easy way to say CloudLinux. From my other issue uname -r gives: 4.18.0-513.18.1.lve.2.el8.x86_64 which if you called it and split the string would break down into:

lve = CloudLinux el5, el6, el7, el8 = Version 5,6,7,8

This should work if you can call uname on a particular server, else go with option 1 as a backup. However, from the links below this may not work on version 9 which does away with the lve string 🤡

It's weird how the fans are really excited that you can only tell what version you are on from decoding the uname.

Links: Converting CentOS server with cPanel List of CloudLinux versions with corresponding uname results??!! Why CloudLinux OS 9 doesn't have kernel with LVE in its name? CloudLinux is installed properly? How to select the correct kernel to boot from on CloudLinux what version am i running and how do i upgrade Troubleshooting the Absence of LVE in CloudLinux OS 9 Kernel Name

HorlogeSkynet commented 3 months ago

(We already process uname output by default, so it's OK modulo (security) concerns raised in #311 and #328, but that's a totally different issue :slightly_smiling_face:)

Okay, so I've processed most of your pointers, below is a quick script (RFC), that I'll try to implement if it looks good to you :

release_components = uname_release.split(".")
if len(release_components) > 1 and release_components[-2].startswith("el"):
    return {
        "id": "cloudlinux",
        "name": "CloudLinux",
        # strip "el" prefix and replace underscores by dots
        "version": release_components[-2][2:].replace("_", "."),
    }

This code would land in current _parse_uname_content function (which requires a little tweak to deal with "Linux" special case...).

Note : the implementation is (as always) opinionated : I deliberately skipped the part with lve kernel module indication, which, as you pointed out, misses from CloudLinux 9+. I consider el* as penultimate release component "enough specific" to identify CL. I guess this conflicts with RHEL/CentOS :cry:

Tell me what you think ! See you :wave:

EDIT : I've come up with an implementation, I try to add missing tests before submitting it.

HorlogeSkynet commented 3 months ago

Hello @NebularNerd ! Could you please take a look/give a try to #369 RFC ? :slightly_smiling_face: Thanks bye :wave:

NebularNerd commented 3 months ago

Tested #369 on the GoDaddy 🥔 server and we have a winner 🎉

{
    "codename": "",
    "id": "cloudlinux",
    "like": "",
    "version": "8",
    "version_parts": {
        "build_number": "",
        "major": "8",
        "minor": ""
    }
}
HorlogeSkynet commented 3 months ago

Awesome, we only have to wait for @python-distro/maintainers to properly review #369 then :slightly_smiling_face: Many thanks for your time and resources Andy, see you :wave: