Remove dialects we cannot maintain

[masatake]

• [ ] aix
• [ ] du
• [ ] hpux
• [ ] n+os
• [ ] sun

@filbo will maintain osr and uw.

[floam]

I might suggest leaving some of these for now. Perhaps Sun might be one you could spare. It intuitively feels like there are still regular people doing active OpenSolaris development as https://illumos.org for example. You would not need to officially support the platform.

By leaving them, perhaps broken, perhaps working, unknown, it gives random people the opportunity to attempt to use it on e.g. Solaris, or AIX, and while it may be buggy or it may not compile, the project will benefit from getting a good bug report explaining what is wrong. It is okay to leave these bug reports "open". One can perhaps assign a milestone like as "orphaned" or "needs a hero" or a vague "future". Perhaps someone else will come along and submit a fix a few months later. This might be better than giving the impression that lsof has never supported the platform, where no activity will occur here.

Thoughts?

[masatake]

Yes, you are correct. If we keep the source code for the platforms, we have chances of getting bug reports about these platforms. Especially, if the bug is about a security issue, people can consider this project/repository is the first contact point. This is what I don't want.i I don't want to receive bug reports for what we can do nothing.

Instead of making people using lsof on these platforms misrecognize as if there is a viable project, making them recognize there is no such project is better.

I will remove the code as far as there is no volunteer.

This might be better than giving the impression that lsof has never supported the platform, where no activity will occur here.

Giving the impression that lsof has never supported the platform if there is no volunteer for the platform is that I would like to let people know through this issue.

If lsof is a new software/project, acting to get more users may be important. However, lsof is there for a long time.

[floam]

Fair enough. In practice of course I think you are not going to find there are very many people actually compiling new releases of lsof on any of those platforms.

I just read about your plans for testing - I'm glad your team is going to have continuous integration happening for Linux, different BSDs, and macOS. That is basically what we've covered for our project, a command line shell (https://github.com/fish-shell/fish-shell), and it really has paid off big time for everyone. It's easy for me to ensure a good Linux experience while I develop on my Mac, while a team member on Linux often addresses NetBSD issues for example. We just do not like commits that break tests.

[filbo]

I have successfully built, tested (cd tests; make; make opt), and run 4.93.2 'osr' target on SCO OpenServer 5.0.6. There is a minor build breakage (doesn't generate man page), easily handled.

People are still using every one of the dialects touched by lsof. Nothing ever really dies in IT space. It just becomes obscure.

[krytarowski]

While there, NetBSD and OpenBSD could be separated. They are currently maintained under a cryptic name n+obsd.

[masatake]

Do you want to maintain the dialects?

Before saying yes, I would like you to read the source code of lsof and read https://github.com/lsof-org/lsof/pull/70 .

To develop lsof, you need the knowledge of the kernel of the target OS. If the kernel is updated, you have to catch up it. You may have to use ifdefs to cathcing up the updated kAPI. I found an interesting idea to measure the difficulties: '#if' dencity:

[yamato@slave]~/var/lsof/dialects% for x in *; do echo -n "$x": ; echo $(grep -r '# *if' $x | wc -l)/$(find $x -type f  | xargs cat | wc -l) = $(perl -e "print $(grep -r '# *if' $x | wc -l) / $(find $x -type f  | xargs cat | wc -l)");  done
aix:324/7088 = 0.0457110609480813
darwin:208/8874 = 0.023439260761776
du:176/5563 = 0.0316376056084846
freebsd:357/5969 = 0.0598090132350477
hpux:380/11421 = 0.0332720427283075
linux:200/12118 = 0.0165043736590196
n+obsd:228/4390 = 0.0519362186788155
n+os:66/3446 = 0.0191526407428903
osr:123/5032 = 0.0244435612082671
sun:555/15273 = 0.036338636810057
uw:222/6252 = 0.0355086372360845

Are you ready to maintain K&R C + too many ifdefs source code?

---

.:87367/53677169 = 0.00162763800005921

This value is for linux kernel itself.

.:53023/9661998 = 0.00548778834357035

This is value is for freebsd kernel(sys) itself.

If ifdef dentistry of kernel K is X, ifdef density of lsof for K is 10 * X. I would like someone to evaluate my hypothesis on darwing.

[masatake]

@jrybar-rh, @karelzak, I would like to know about https://src.fedoraproject.org/rpms/lsof/blob/master/f/upstream2downstream.sh .

Do you know which dialects have the problem with copyrights for some UN*Xes ? I would like to remove the from the git repo of this project.

• [ ] aix
• [ ] du
• [ ] hpux
• [ ] n+os
• [ ] osr
• [ ] sun
• [ ] uw

[krytarowski]

@masatake NetBSD has users and lsof is in pkgsrc.. however we plan to add a better userspace interface to list open file descriptors in a process. Before materializing that, it's better to hold on.

Regarding other kernels, I have nothing to say.

[masatake]

@masatake NetBSD has users and lsof is in pkgsrc.. however we plan to add a better userspace interface to list open file descriptors in a process. Before materializing that, it's better to hold on.

Thank you for the great input. The new interface may reduce the "#ifdef"s ! I have no short plane to remove OSS kernel. What I would like to delete first are proprietary ones.

[krytarowski]

Just go ahead. If there is interest, users of these OSs will reintroduce the support with updated code.

[karelzak]

@jrybar-rh, @karelzak, I would like to know about https://src.fedoraproject.org/rpms/lsof/blob/master/f/upstream2downstream.sh .

Do you know which dialects have the problem with copyrights for some UN*Xes ?

I don't remember which dialect (it's 13 years ago). Frankly, everything else than Linux is irrelevant for Fedora/RHEL, so the best for us was to remove all non-Linux stuff rather than try to dance with lawyers ;-)

It's a good idea to start with only code you can maintain and test. IMHO everything else than Linux, *BSD and GNU/Hurd is waste of time in these days -- the other users can still use the original Vic's tarballs.

(and big respect to Vic Abell! -- it's amazing that he was able to contribute to open-source in age ~80)

[masatake]

@karelzak, thank you for the comment.

[krytarowski]

I agree with karelzak. BTW. Vic Abell is a BSD developer.

[masatake]

@krytarowski, are there any documents or discussion logs about the better userspace interface to list open file descriptors in a process of NetBSD? I would like to study the interface.

[krytarowski]

Basically, please check FreeBSD's kinfo_getfile():

https://www.freebsd.org/cgi/man.cgi?query=kinfo_getfile&apropos=0&sektion=0&manpath=FreeBSD+12.1-RELEASE+and+Ports&arch=default&format=html

It wraps internally sysctl: KERN_PROC_FILEDESC.

We should get something similar in NetBSD.

[masatake]

Thank you!

[krytarowski]

If you could switch FreeBSD to it, it would be a good reference for NetBSD later. Certainly other BSDs will catch up.

[masatake]

lsfd, I'm thinking about developing a new command replacing lsof PARTIALLY.

[krytarowski]

Do you mean a successor of lsof?

[masatake]

YES. I was thinking about submitting lsfd.c as a new command to https://github.com/karelzak/util-linux . However, now you let me know the new interface of BSDs. (I have written nothing yet. Just an idea.)

[krytarowski]

If you will end up with a tool in util-linux it is fine, but it will be Linux-only.

[masatake]

Yes, that's the matter.

[filbo]

Please don't remove those commercial dialects. They are still in use.

Vic Abell was excruciatingly careful to get written permission from each of the vendors whose copyright-marked headers he included; in most cases (I think 'all' but I am less certain of this), the headers he provided were ones which the vendor had intended to include in the kernel build subsystems of their products, but had been left out of some release at some point. Since lsof was and is intended to be able to build on any release of any OS it's 'familiar with', it was necessary to provide those missing headers to address those particular releases where they vendor had forgotten them.

If you go around removing dialects you are not in fact maintaining lsof. You're maintaining some decayed subset of it.

Do as Vic did: mark which dialects are tested and supported. Leave the others alone. Accept patches for them as long as the patches are not destructive to supported dialects.

Regarding an lsfd command, are you intending to support all of the command-line flags of full lsof? In that case you are either rewriting a whole lot of parsing etc., or starting with actual lsof source and removing everything non-Linux. In these three cases:

(1) writing your own code to support existing command line args & output formats: you're doing a bunch of extra work and will struggle to achieve strong compatibility anyway

(2) stripping away all non-Linux: you're making a bad fork of a project. You will miss bug fixes from mainline lsof, mainline lsof will miss fixes from you, command line args will become incompatible over time, etc.

(3) writing a new utility with different args, output, etc.: every one of those flags is there because someone had a use for it. You're embarking on the first steps of a long and painful journey

Obviously I recommend against it :)

This is a cross-Unix utility. Please don't break that aspect of it.

[filbo]

Vic Abell was not just the 'maintainer' of lsof. He was (is) its original author.

[masatake]

@filbo, I invited you to the team. As far as you are here, we should not remove osr and uw.

[xosevp]

State of operating systems.

dialect	OS	state	notes
aix	AIX	current
darwin	macOS	current
du	Tru64 UNIX	dead
freebsd	FreeBSD	current
hpux	HP-UX	frozen	EOL 2026
linux	Linux	current
n+obsd	NetBSD + OpenBSD	current
n+os	NeXTSTEP + OPENSTEP	dead
osr	OpenServer	frozen
sun	Solaris	current	EOL 2035
uw	UnixWare	frozen

[karelzak]

(1) writing your own code to support existing command line args & output formats: you're doing a bunch of extra work and will struggle to achieve strong compatibility anyway

Frankly, if one day anyone will start from scratch then it will be probably better to write a new tool without care about backward compatibility. It would be better to have a simple and lightweight tool than try to reimplement lsof. I can imagine lsfd based on libsmartcols for output formating (like lsblk, findmnt, etc) with basic filtering features like file type, name, directory, or user.

[masatake]

I can imagine lsfd based on libsmartcols for output formating (like lsblk, findmnt, etc) with basic filtering features like file type, name, directory, or user.

YES. I can imagine a tool based on libsmartcols, too.

What I would like to do in lsfd are:

• using C99 at least,
• focusing only on Linux,
• printing the list with libsmartcols,
• providing long options,
• using multiple threads when gathering fd information if it improves performance,
• providing namespace related features,
• providing a sophisticated filter engine working in gathering-information-time,
• ...

[masatake]

@karelzak, I wrote an version 0.0.0 of lsfd based on libsmartcols. As I expected the library is quite powerful. I got json output without cost.

[yamato@control]~/var/util-linux% ./lsfd --help
./lsfd --help

Usage:
 lt-lsfd [options]

Options:
 -J, --json            use JSON output format
 -n, --noheadings      don't print headings
 -o, --output <list>   output columns
 -r, --raw             use raw output format

 -h, --help            display this help
 -V, --version         display version

Available output columns:
         PID  PID of the process opening the file
          FD  file descriptor for the file
        NAME  name of the file

For more details see lsfd(1).
[yamato@control]~/var/util-linux% sudo ./lsfd | head
sudo ./lsfd | head
    PID    FD NAME
      1     0 /dev/null
      1     1 /dev/null
      1     2 /dev/null
      1     3 /dev/kmsg
      1     4 anon_inode:[eventpoll]
      1     5 anon_inode:[signalfd]
      1     6 anon_inode:inotify
      1     7 /sys/fs/cgroup/unified
      1     8 anon_inode:[timerfd]
[yamato@control]~/var/util-linux% sudo ./lsfd --json | head
sudo ./lsfd --json | head
{
   "lsfd": [
      {
         "pid": 1,
         "fd": 0,
         "name": "/dev/null"
      },{
         "pid": 1,
         "fd": 1,
         "name": "/dev/null"
[yamato@control]~/var/util-linux% 

I will polish the code for submitting to the mailing list of util-linux.

[masatake]

I will close this issue. Instead of removing the dialects from lsof, I decide to develop lsfd, a lsof alike new tool speciflized to Linux. I will try to merge lsfd to util-linux.

[masatake]

The development version of lsfd is available at https://github.com/masatake/util-linux. See lsfd branch.

[floam]

Thank you for allowing this project to maintain lsof, @masatake.

[masatake]

Instead of "FD" field, I introducedd ASSOC field. The field represents a process and file objects.

# ./lsfd
COMMAND     PID            USER  ASSOC TYPE DEVICE      INODE NAME
systemd       1            root    cwd  DIR  253,0          2 /
systemd       1            root    exe  REG  253,0    1850829 /usr/lib/systemd/systemd
systemd       1            root   root  DIR  253,0          2 /
systemd       1            root cgroup  REG    0,4 4026531835 cgroup:[4026531835]
systemd       1            root    ipc  REG    0,4 4026531839 ipc:[4026531839]
systemd       1            root    mnt  REG    0,4 4026531840 mnt:[4026531840]
systemd       1            root    net  REG    0,4 4026532000 net:[4026532000]
systemd       1            root    pid  REG    0,4 4026531836 pid:[4026531836]
systemd       1            root  pid4c  REG    0,4 4026531836 pid:[4026531836]
systemd       1            root   time  REG    0,4 4026531834 time:[4026531834]
systemd       1            root time4c  REG    0,4 4026531834 time:[4026531834]
systemd       1            root   user  REG    0,4 4026531837 user:[4026531837]
systemd       1            root    uts  REG    0,4 4026531838 uts:[4026531838]
systemd       1            root    mem  REG  253,0    1850829 /usr/lib/systemd/systemd
systemd       1            root    mem  REG  253,0    1850829 /usr/lib/systemd/systemd
systemd       1            root    mem  REG  253,0    1850829 /usr/lib/systemd/systemd
systemd       1            root    mem  REG  253,0    1850829 /usr/lib/systemd/systemd
...
systemd       1            root    mem  REG  253,0    1838886 /usr/lib64/ld-2.32.so
systemd       1            root      0  CHR    1,3          4 /dev/null
systemd       1            root      1  CHR    1,3          4 /dev/null
systemd       1            root      2  CHR    1,3          4 /dev/null
systemd       1            root      3  CHR   1,11         10 /dev/kmsg
systemd       1            root      4 UNKN   0,13      13420 anon_inode:[eventpoll]
systemd       1            root      5 UNKN   0,13      13420 anon_inode:[signalfd]
systemd       1            root      6 UNKN   0,13      13420 anon_inode:inotify
systemd       1            root      7  DIR   0,27          1 /sys/fs/cgroup