./build is very inflexible

[keenerd]

Trying to package Fexl for Arch Linux. Running into a slew of issues.

• You can't build as an unprivileged user.
• There is no equivalent to make DESTDIR=...
• It does not create /usr/bin or /usr/share if these locations do not already exist.

I've hacked around this using a unionFS sandbox so that ./build thinks it is writing to the root filesystem.

You can see the attempts at trying to fix these in https://aur.archlinux.org/packages/fe/fexl-git/PKGBUILD

But even after after all this stuff, the REPL does not actually work.

[chkoreff]

Thanks, I'll probably have to start testing on many different systems. I only develop on Ubuntu, which I use for all my daily work.

I have no idea why you wouldn't be able to build as an unprivileged user. You certainly can't install as an unprivileged user, but a local build should be simply:

./build

All the obj, lib, and bin directories are created right in your local directory, owned by you.

You are correct that there is no equivalent to "make DESTDIR". At the moment, Fexl is highly "opinionated" about installing things in the standard places only. The README specifies that the installation goes in " /usr/bin/fexl, /usr/lib/fexl, and /usr/share/fexl. I also have a note in the README which says: "If you need to install into a different place, change the dir_install variable inside the build script."

That's the state of affairs at the moment. I'll consider a "dest" option. So for example if you specified a destination of "/X" instead of the standard "usr", the files would go in /X/bin/fexl, /X/lib/fexl, and /X/share/fexl. Keep in mind that my use of bin, lib., and share are in accordance with well-documented standards which I could cite. If you have a better suggestion for how dest might work, I'd appreciate it.

On your third point, wow I never considered that /usr/bin, /usr/lib, or /usr/share might not exist whatsoever! Then again, you are on Arch, which is quite minimalist, unlike me on maximalist Ubuntu, so that makes sense. I pop in a couple of mkdirs for you -- or if you've done it, please send me your build script.

Now when you say " the REPL does not actually work", that baffles me because there is no "REPL" as such. You run Fexl from the command line, giving it a whole program, which it then runs. There's no automatic built-in "Read Eval Print Loop" as in Lisp. Fexl simply does what you tell it to do. It's like C in that respect -- it's just a programming language, not an "environment".

[chkoreff]

I noticed this in your notes: ""Fexl has a convoluted build process that requires writing to your /".

Two things. Convoluted? Here's a fresh install I just did on my machine:

git clone git@github.com:chkoreff/Fexl.git
cd Fexl
./build install

Then just for fun I ran fexl with no script name, and typed print "hello" and Ctrl-D, and it said hello to me.

Obviously you had a different experience, which just goes to show how radically different one Linux can be from another. As far as I can tell, it just needs to do a mkdir on the directories in /usr for minimalist systems that don't have those.

Second point. You say that Fexl "requires writing to your /". Yes, if you want to install something on your system in the standard, well-documented locations for installed software, that's where it goes -- in /usr. But check this out -- Fexl does not require writing to / if you just do a local build. Let me show you one from scratch again:

git clone git@github.com:chkoreff/Fexl.git
cd Fexl
./build

Now to run my locally installed version, I just type bin/fexl to run it. Or you can run it from anywhere using a full path, or put it in your search path. If you don't like the source code laying around there, rm -rf src.

Or, you could do what the README suggests and edit the dir_install variable in the build script. It's currently set to "/usr", but you can change it to "~/fexl" if you prefer. As I mentioned, I may add a "dest" option that lets you do this without editing the script. But I have very carefully researched the standards for installed software, and /usr/bin, /usr/lib, and /usr/share are excellent defaults.

[keenerd]

./build install works on Arch too. But we never want to install something directly to root without going through the package manager.

So the problem is that Fexl is unpackage-albe. It assumes that wherever the files end up, that is the final library path. With make, it is standard practice to

./configure --prefix=/usr
make
make DESTDIR=...

This would install to DESTDIR (as an unprivileged user) but if you ran strings on the Fexl binary you'd see it reference /usr/lib/fexl/libfexl.so instead of ~/fexl/ or wherever it was built.

Similarly, those missing directories exist on Arch - but not when you are installing to the destdir.

The convoluted part is that it needs root privileges for the build at all. Everyone else builds as a user and then raises privileges to copy the files into location.

[chkoreff]

Yep, I'm on it right now. Personally I've only had two use models in mind (1) do a local "./build" for testing things, and then finally (2) do a "./build install" to install things in the One and Only System Directory (i.e. "/usr").

I'm just now adding an option that would let me say something like:

./build install in ~/myfexl

[chkoreff]

OK, check out the new build script. You should be good to go now.

To install somewhere other than /usr, you can say:

./build install in ~/myfexl

To uninstall, say:

./build uninstall in ~/myfexl

You can still install in the default /usr directory if you're an administrator, but now that I've removed "sudo" from the build script, you must do it explicitly, e.g.:

sudo ./build install
sudo ./build uninstall

Let me know if this closes the issue for you, and if there's anything else I can do!

Best Regards, Patrick

[keenerd]

Thanks for the quick patching.

So this looks like a step in a good direction, but it fixes one problem at the cost of another. I agree that /usr is the one true install path, but for packaging purposes you don't write files outside the build directory.

The normal ./build install puts the files in inconvenient places (/usr) but links them properly. ./build [in] puts the files someplace convenient for packaging but links against the local build dir instead of the install target (/usr).

So ./install in /path/foo puts the files where I'd like them, it still links to /path/foo/lib/fexl/libfexl.so. For packaging you want to control the linking path (--prefix) and the installation path (destdir) separately.

For the makefile analogy, ./install in /path/foo is equivalent to ./configure --prefix=/path/foo and no equivalent to destdir.

[keenerd]

For now I am just patching the library path in the binary.

The best way to fix it would be to change line 298 to run "gcc $file_o -o $file_x -L"$dir_lib" -lfexl"

[chkoreff]

On my system, doing a build works properly in any directory "X", linking with the library in "X/lib/fexl/libfexl.so'. It does not link against the local build dir as you say.

For example, I install like this:

./build install in ~/myfexl

And then the output of:

strings ~/myfexl/bin/fexl | grep libfexl

is:

/home/patrick/myfexl/lib/fexl/libfexl.so

And when I do the strings on /usr/bin/fexl, I see, as expected:

/usr/lib/fexl/libfexl.so

In both cases, the executable "X/bin/fexl" is correctly linked to the library "X/lib/fexl/libfexl.so".

Thanks for the suggestion about line 298, I'll try that. Evidently there's an incompatibility between the linker on my Ubuntu system and the linker on your Arch system.

[chkoreff]

OK, I tried your line 298 and it doesn't work on my Ubuntu system. Here's the line I used:

run "gcc $file_o -o $file_x -L"$dir_lib" -lfexl"

And here's me doing the build and seeing what comes up in strings:

patrick@laptop:~/project/fexl$ ./build clean quiet install in ~/myfexl; strings ~/myfexl/bin/fexl | grep libfexl
libfexl.so

Naturally it doesn't work when I try to run it:

patrick@laptop:~/project/fexl$ ~/myfexl/bin/fexl 
/home/patrick/myfexl/bin/fexl: error while loading shared libraries: libfexl.so: cannot open shared object file: No such file or directory

Let me try something else that might be compatible between Ubuntu and Arch ...

[chkoreff]

OK, I just tried a line 298 that's a compromise between what I had and what you suggested:

run "gcc $file_o -o $file_x -L"$dir_lib" $dir_lib/libfexl.so"

It looks good on the build and strings:

patrick@laptop:~/project/fexl$ ./build clean quiet install in ~/myfexl; strings ~/myfexl/bin/fexl | grep libfexl
/home/patrick/myfexl/lib/fexl/libfexl.so

And it runs properly:

patrick@laptop:~/project/fexl$ ~/myfexl/bin/fexl 
print "hello";nl;
hello

So, can you try my line 298 on your Arch system and let me know if it works? There's gotta be something that works on both, I just know it!

Thanks, Patrick

[chkoreff]

Yikes, I just read a comment of yours again, and it suggests that when installing to directory "X", you actually don't want it to link to the library in "X/lib/fexl". Now all bets are off, and I am utterly confused, because I wanted installation to follow the simple rule that when you installed in "X", it would create all of its output files (bin, lib, and share) underneath "X".

But your comment above says, in the nature of a complaint: "So ./install in /path/foo puts the files where I'd like them, it still links to /path/foo/lib/fexl/libfexl.so." As if that's a BAD thing! :(

The last thing I want to do is go down the road of so many murky installation processes involving autoconfigure, with endless options about where to put this, that, and the other. I just want one drop-dead simple "./build install in X", which follows one drop-dead simple rule: that the output files will go underneath "X" in the bin, lib, and share directories.

Now you say "For packaging you want to control the linking path (--prefix) and the installation path (destdir) separately.", and the blood drains from my face as I consider the enormous amounts of life energy I don't want to spend on endless options and build platforms. I just can't afford it at this point in my life. I had hoped, apparently wrongly, that there would be enough compatibility among the various Linuxes that I wouldn't have to worry too much about it, and I felt very confident that the simple rule about "X" would be just fine.

Evidently not. So at this point, I am officially out of my league. I certainly appreciate your comments and suggestions, but I just can't summon the will to add yet another configuration option.

[chkoreff]

Cooling off just a bit, here's the thing. When I say install in X1, and then install in X2, I expect and want X1/bin/fexl to link to X1/lib/fexl, and I expect and want X2/bin/fexl to link to X2/lib/fexl. That's a GOOD thing. I certainly wouldn't want both to link to X1, or either to link to /usr/lib, yadda yadda yadda. The whole point is for me to try different installations which are totally independent of each other. I don't mind having multiple copies of the library, any more than I mind having multiple copies of the executable itself, or multiple copies of the share directories. They're all part and parcel of that specific installation in X1, X2, /usr, or wherever else. I don't want complex, inscrutable cross-links all over my file system.

But that's just me. :)

Now, on the subject of "packaging" as you call it, I certainly have no idea how to create things that will install with "apt-get", or "deb" or whatever other programs are used on various Linuxes -- I just don't know anything about it. So if you have any suggestions there, I'd appreciate it.

[chkoreff]

You know, one thing I don't like about saying "sudo ./build install" is that it builds the obj/xxx.o files with root permissions. Subsequently I can permission errors if I do anything as an ordinary user that requires deleting or updating one of those object files -- e.g. merely updating a .c file and running ./build. So I pretty much have to say "sudo ./build erase" and start fresh.

In the original build script, I'd call sudo inside the script when you were installing to /usr. That way your object files would be created with yourself as owner instead of root.

Now that you can install to places other than /usr, I can't do that anymore. I was thinking of perhaps always deleting the local obj and bin directories after an install, which at least wouldn't leave local files owned by root, but would be a pain because every install would force a clean build. Ugh, no good options here.

[falconindy]

No offense, but you're driving yourself mad explicitly because you opted to not use something like autotools, and you're reinventing it with what appears to be a neophyte level of understanding of compilation and shell.

You need to leverage the linker. Embedding absolute paths in a soname dependency is insane. If you desperately want the linker to use a library within a constrained path, this is why rpath exists (but please don't use that either). Personally, I'd recommend creating a static lib and linking against that, even if only for some sort of local test environment (short of that, use LD_PRELOAD or LD_LIBRARY_PATH).

If you're going to provide a shared library, please create it according to some semblance of standards (i.e. the 99% of the Linux world), otherwise there isn't a whole lot of point to providing a shared library. I don't know what version of Ubuntu you're using, but I suspect you've been trapped into using a version with Multiarch, which has its own slew of subtle problems. There still aren't enough major differences between our toolchains for this to be as big of a deal as you're making it out to be.

[chkoreff]

Simple case in point:

sudo ./build install
sudo -k   # so I'm no longer root
touch src/fexl.c
./build
patrick@laptop:~/project/fexl$ ./build
gcc -c -Wall -Werror -O3 -fPIC src/fexl.c -o obj/fexl.o
Assembler messages:
Fatal error: can't create obj/fexl.o: Permission denied

[chkoreff]

When I do multiple installations, I do it for this reason: to try out completely independent and self-contained programs, each bundled with its own version of bin, lib, and share.

So when I install into "X", for any value of X, I know that I have these files:

X/bin/fexl
X/lib/fexl/libfexl.so
X/share/fexl/

I further know that the following statements are true:

X/bin/fexl uses the C library X/lib/fexl/libfexl.so.
X/bin/fexl uses the Fexl library X/share/fexl.

To underscore my point, I know that those are true for all values of X, including ~/myfexl_v1, ~/myfexl_v2, /usr, /way_up_top, etc. etc. I actually don't mind that each of those four installations uses four distinct versions of lib/fexl and share/fexl -- on the contrary, it's highly likely that those libraries are in fact different, otherwise I wouldn't have bothered doing four separate installations!

I take no offence at your suggestions that this is due to a neophyte understanding or lack of adherence to standards. All of that may be true. However, those are not the reasons for my doing it this way. The reason I do it this way is far simpler: it's because that's how I want it to be. The way I do it now achieves that. I might someday be convinced that I shouldn't want it that way, but that day is not now.

You mention that this does not give you the benefits of a truly shared library. That's kind of a good point, and it really brings into question why I bothered with a shared library at all. Why not just link everything into a single executable, or use a non-shared library? Am I really trying to provide a library which other people can link into their own projects, like the OpenSSL library? What does that even mean for Fexl? I see no value in it.

I think the only reason I bothered with a shared library was to limit the amount of memory used by multiple instances of the fexl executable running on a single machine. If I linked everything into one executable, I wasn't entirely sure if multiple instances of bin/fexl would be loaded only once in memory. But when I researched shared libraries, I felt assured that multiple instances of bin/fexl would definitely load the shared library only once in memory.

[chkoreff]

Oh wait -- DUH, now I remember why I bothered to use a shared library. It's so that Fexl programs can dynamically call functions! If I linked all my reduce_X functions into the executable, there's no way I can call "dlsym" to grab a function handle given a string X. (See my src/dynamic.c file.)

So, at least I don't have to question why I use a shared library anymore. My choice had nothing to do with sharing per se -- it was all about the ability to call dlsym.

[chkoreff]

I wrote: "You know, one thing I don't like about saying "sudo ./build install" is that it builds the obj/xxx.o files with root permissions."

Simple solution for that, going into build script now:

# Now, in case you're running under sudo, change the permissions on the local
# bin and obj directories back to the underlying user.  Otherwise you'd have
# files owned by root in your local directory, which is annoying.
if [ $install -eq 1 ]; then
    chown -R $SUDO_USER:$SUDO_USER bin
    chown -R $SUDO_USER:$SUDO_USER obj
fi

[chkoreff]

OK, I did roughly that, only better:

https://github.com/chkoreff/Fexl/commit/8e5aa604c357015a906a887acd2a39f46cd04c9f

That eliminates all possibility of having locally created files owned by root.

[chkoreff]

I've also updated the README https://github.com/chkoreff/Fexl

[chkoreff]

falconindy wrote "No offense, but you're driving yourself mad"

Only for a moment. I'm very happy with the state of affairs now, and utterly at peace. I updated the README with an explanation for my rationale for supporting completely independent, self-contained installations X1, X2, X3, etc. I actually want it that way!

[falconindy]

That's fine. I'll only point out that it's unfortunate that you've designed a limitation that makes it non-trivial for a distro to package your software as is.

[chkoreff]

I'm willing to consider what you say there, but since I know absolutely nothing about how distros "package" things, I have no way to address it with concrete action at the moment. I will have to rely on the skills of others for now. If you devise a way to hack my install process in a way which enables a distro to package it more easily, then please let me know and I'll incorporate your suggestions.

Again, I currently know nothing about "packaging", so just for now I'm content with the ease of installing from source with:

git clone git@github.com:chkoreff/Fexl.git
cd Fexl
sudo ./build install

I'm also very happy with the ability to create multiple versions easily, e.g.:

./build install ~/fexl_v1
# ... change a bunch of files, affecting the behaviour of
# everything, including bin, lib, and share ...

./build install ~/fexl_v2

And then being able to run both of those independently:

~/fexl_v1/bin/fexl
~/fexl_v2/bin/fexl

I love all of that. However, if I'm hearing everyone correctly, to better support "packaging" I would need NOT to link the X/bin/fexl directly with X/lib/fexl/libfexl.so, instead enabling the libfexl.so to be "somewhere else" -- exactly where I'm not sure.

Also, keep in mind that I also "link" the X/bin/fexl directly with X/share/fexl (i.e. the library written in Fexl). I put "link" in quotes because it's not the linker that establishes that hard tie, but rather the "base" directory which the build script writes into obj/base_install.c. Note that this also allows me to support "base_path" as a built-in Fexl function (defined in src/meta.c).

So, the upshot is that I need to know a highly specific concrete action which would better support your packaging efforts. I'm willing to do the work, but someone else has to provide the guidance. I also need to be able to make this change without forfeiting any of the aforementioned features that I love.

[chkoreff]

To avoid getting caught up in generalities, I'd like to focus on a few questions which I think are highly salient. After installing fexl using your packaging method (i.e. not from source via github), what would be the output of the following commands?

First:

$ which fexl

Second:

$ strings `which fexl` | grep libfexl

Third:

$ echo "base_path \x print x;nl" | fexl

I believe if I knew the specific three answers you envision there, I would gain much insight into what you'd like me to do.

Incidentally, here are the current answers to those three questions on my own system:

$ which fexl
/usr/bin/fexl

$ strings `which fexl` | grep libfexl
/usr/lib/fexl/libfexl.so

$ echo "base_path \x print x;nl" | fexl
/usr

I suppose you would prefer that the answer to the second question be:

$ strings `which fexl` | grep libfexl
libfexl.so

If that's the only difference you want to see, then I will simply add an option not to link the executable to the specific (C) library path. That would be easy. I would however still "link" to the specific Fexl library /usr/share/fexl, since I know of no other way to establish that association.

[falconindy]

So, enough moaning from my side -- just for fun, I rewrote your build system with make. I think it still retains everything you want, though it does not keep the same directory structure when installing locally as compared to a root install (I don't think this really matters). It also makes a point of always doing the linking the Right Way™. I created an 'install-local' targets to facilitate this, and it includes a toplevel "fexl" 2 line shell script which uses LD_LIBRARY_PATH to bootstrap the binary with the correct shared lib.

I started rewriting some of the doc in the README about how to install, but stopped at the 'how to build' section.

Feel free to clone my tree and poke around.

[chkoreff]

Thanks, I'll play around with that and see if I can incorporate any code or concepts from it. I'm not thrilled with having the actual fexl program be a shell script, but I'm keeping an open mind.

One small note, you mentioned the "hackish" char star called "base", which you have converted to a processor directive. I implemented base as a char star specifically because I am assiduously and consciously avoiding the use of preprocessor directives. It's just a challenge I've always set for myself.

The bigger problem is that when I run "make" on your code, it dies with an error on my Ubuntu machine:

patrick@laptop:~/tmp/Fexl$ make
make -C src PREFIX=/usr
make[1]: Entering directory `/home/patrick/tmp/Fexl/src'
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o basic.o basic.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o buf.o buf.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o die.o die.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o double.o double.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o dynamic.o dynamic.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o fexl.o fexl.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o file.o file.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o io.o io.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o long.o long.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o memory.o memory.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o meta.o meta.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o parse.o parse.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o parse_file.o parse_file.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o parse_string.o parse_string.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o resolve.o resolve.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o run.o run.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o string.o string.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o sym.o sym.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o timer.o timer.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o type.o type.c
cc -fPIC -DBASE_PATH=\"/usr\"    -c -o value.o value.c
test -d ../bin || mkdir ../bin
cc -shared -fPIC -DBASE_PATH=\"/usr\"   basic.o buf.o die.o double.o dynamic.o fexl.o file.o io.o long.o memory.o meta.o parse.o parse_file.o parse_string.o resolve.o run.o string.o sym.o timer.o type.o value.o -o ../bin/libfexl.so
make[1]: Leaving directory `/home/patrick/tmp/Fexl/src'
test -d bin || mkdir bin
cc -DBASE_PATH=\"/usr\"  src/fexl.c -ldl  -Lbin/ -lfexl -o bin/fexl
bin//libfexl.so: undefined reference to `dlsym'
collect2: ld returned 1 exit status
make: *** [bin/fexl] Error 1

But I'm not sure what you'd need to change to make "dlsym" available.

[falconindy]

Reorder the compiler arguments is what I'd need to do ;) That's fixed on my branch.

Not sure what your opposition is to using preprocessor. Yes, it's true that it can easily be abused, but this is a very straightforward use case where it simplifies linkage, as you're not creating an entirely isolated translation unit which only serves to reference a path. In fact, using the preprocesor here can only help.

The shell script adds a a trifling amount of overhead, and no new PIDs.

[keenerd]

Just a little bit more about why we'd like to see these changes and how they related to (binary) packaging. For the rest of this, when I say packaging I will always mean binary packages.

No matter what the system, an unprivileged user will do the build. The files from the build are tarballed up (with some meta data) and transfered to the computer where they will be installed. The install process consists of adding the new meta data into the package manager and copying the files to root.

Many people will never consider using sudo ./build install because this sprinkles files around the hard drive that are untracked by the package manager. Similarly, there is a common distrust of "uninstall" functions. Maybe in the future you'll put files someplace else. The original locations might not be removed by your custom uninstall. So we let the package manager handle uninstalling.

Back to packaging, once you consider how a package is made the split between --prefix and destdir makes quite a lot of sense. Destdir is where your unprivileged user doing the build has write permissions. Prefix is where the package will eventually be installed. The files need to be copied to somewhere under home so they can be tarballed into a package. But the same files need to be executable when they are extracted to root. For all the complexity of autotools, it is very good at this sort of thing.

If you are going for multiple versions of the same program and don't want a launcher script to force the correct library version, don't use shared libraries. Link libfexl statically. It is just 45KB. Even running a hundred instances at once will only waste 4.5MB of ram. The only reason for you to have a shared library is to make it easier for third parties to embed Fexl in their programs. And (in my experience) these third parties want to use static libraries anyway because they don't appreciate the language being updated/changed.

So what to do about the dynamic string eval? Well don't use the dlsym hackery either. If you want to do string lookup, make a table of strings and function pointers. Or a big if/else block. If you say "but there are so many functions and I can't write/maintain such a table" I will suggest using perl/python/whatever to automatically generate the C code for the table from the sources. If you are really concerned about speed, use a trie for O(log(n)) lookup time. Though I imagine with the relatively small number of functions have that even a naive O(n) list search will be much faster than reinvoking dlsym for every call.

Falconindy is known for writing very reliable and clean code - these makefiles are no exception. Even if you don't use them yourself it would be a good idea to keep them around so that people can more easily package your code.

[chkoreff]

Just very quickly on this one point, where you say "don't use shared libraries. Link libfexl statically."

As I mentioned in an earlier post, I must use a shared library because that is the only way a call to "dlsym" can possibly work. That is the only real reason I bothered to use a shared library. I had no other motive in mind.

If memory serves, in my earliest experimentation and research, I established that dlsym does not work with a statically linked library.

[chkoreff]

Getting to your later point, where you say this: "So what to do about the dynamic string eval? Well don't use the dlsym hackery either. If you want to do string lookup, make a table of strings and function pointers."

The answer to that is a big emphatic NO. Here's why.

Currently, when I want to add a new plugin X, I edit a .c file in the src directory and create a function with the name reduce_X. Done. That is all. There's nothing else to do but build and run.

No central "registries" or tables or anything like that. What would be the point? The shared library already contains a look-up table of exactly the nature you describe, and dlsym makes use of that!

And OMG, i can't believe you said this: "much faster than reinvoking dlsym for every call." That statement indicates that you have absolutely no idea how Fexl works. There is only one call to dlsym for each symbol X that you actually use in a Fexl program.

Now come on folks, I appreciate the help and advice -- I really do -- but I'm starting to feel like I'm dealing with knee-jerk reactions here.

[chkoreff]

I'm going to start posting a few issues with falconindy's build here. If necessary, I'll take them over to his fork.

I ran "make", and I notice a few things.

First, please set cflags to include:

-c -Wall -Werror -O3

Second, please make it so that the build does not create .o files inside the src directory. I don't like anything polluting or altering the src directory in any way. Keep the .o's in a separate obj directory.

Third, immediately after doing the make, I tried this:

patrick@laptop:~/tmp/Fexl$ ./fexl
exec: 5: ./bin/fexl: not found

Please make it so I can run the fexl executable locally without installing.

Thanks, I'll keep you posted if I uncover more issues.

[chkoreff]

Next issue, I tried this as instructed in your README:

make PREFIX=$HOME/myfexl
# ... (snip compiler output) ...
make PREFIX=$HOME/myfexl install-local
patrick@laptop:~/tmp/Fexl$ make PREFIX=$HOME/falconfexl install-local
make -C src PREFIX=/home/patrick/falconfexl
make[1]: Entering directory `/home/patrick/tmp/Fexl/src'
make[1]: Nothing to be done for `default'.
make[1]: Leaving directory `/home/patrick/tmp/Fexl/src'
cp -r bin/ /home/patrick/falconfexl/bin
cp: cannot create directory `/home/patrick/falconfexl/bin': No such file or directory
make: *** [install-local] Error 1

Please create the installation directory automatically.

So I went ahead and said "mkdir ~/falconfexl" and did the make again, and it was OK.

Then I tried to run the newly created fexl as follows:

patrick@laptop:~/tmp/Fexl$ ~/falconfexl/fexl 
exec: 5: /home/patrick/falconfexl/bin/fexl: not found

So, same problem as the local build: no bin/fexl is created.

[chkoreff]

Next issue. Please make your build process detect header file dependencies and recompile what is needed accordingly.

For example, if I touch "src/parse.h", I expect parse.c, parse_file.c, and parse_string.c to recompile.

If I touch "src/value.h", I expect almost everything to recompile. I just tried editing value.h in your fork, adding a new field to the value struct. When I ran "make", nothing happened because it thought everything was up-to-date.

Also, please do this in such a way that I don't have to change the Makefile every time I add or delete a new header dependency. My own build process does not require that, so I'm spoiled rotten in that regard.

Thanks, Patrick

[chkoreff]

Hey, I was about to get into my code and use the preprocessor directive for the base path, as falconindy is doing. But then I started thinking about how to avoid stale .o files when building for different target directories, one after another.

That prompted me to think about what's actually happening in falconindy's "fexl" wrapper script, which uses the shell variable $0 to fetch the absolute path of the running executable. That shell variable is beautiful because it works even when you invoke the script indirectly through a $PATH search.

So I thought, why can't I do in the C code exactly what $0 does in the shell code? So I dug a little bit into exactly how the /bin/sh program implements "$0" in the first place. After all, /bin/sh is written in C, and it must have a good way of doing it, so I can just use the same method.

Best I can tell, /bin/sh calls readlink(2) to get that information. Check this out:

http://www.linuxquestions.org/questions/programming-9/how-to-get-the-absolute-path-of-the-running-executable-file-and-current-work-path-911068/

To test it out, I hacked the main routine in fexl.c to do this:

/*TODO remove this code, it's just for testing*/
printf("hello world %s\n", argv[0]);
char buf[1024];
ssize_t len = readlink("/proc/self/exe", buf, 1024);
printf("readlink returns \"%s\" len=%ld\n", buf, len);
exit(22);

Then I set $PATH=~/project/fexl/bin:$PATH, and then simply typed "fexl". Here's the output:

patrick@laptop:~$ fexl
hello world fexl
readlink returns "/home/patrick/project/fexl/bin/fexl" len=35

The readlink returned exactly the right result -- the same thing $0 would return in /bin/sh.

Notice that my "hello world" line just prints argv[0], which demonstrates that argv[0] is completely useless for this task.

So, it looks like I can use readlink and completely avoid the need for any BASE_PATH stuff at all. It's probably very portable, since presumably /bin/sh itself is using it. (I guess if I had the gumption I'd go look at the source code for /bin/sh right now.) This seems analogous to Perl's "FindBin" as well.

Let me know what you think. I'm gonna branch a version that doesn't use BASE_PATH. To implement reduce_base_path in meta.c, I'll have to call readlink there.

Of course, when doing this I'll have to strip off the final leg of the path, as falconindy does using ${0%/*}.

I'll also have to use PATH_MAX or whatever instead of 1024. I think it's defined in limits.h or some such place.

[chkoreff]

Also, readlink does not append a null byte to the result, and truncates if the buffer is too small. Unfortunately it does not report the truncation condition by returning -1 or setting errno, so I have no way of knowing if I need to use a larger buffer.

This reminds me of my notes on how to implement getcwd in Fexl. In my research, I discovered this technique in "man getcwd":

If the length of the absolute pathname of the current working directory,
including the terminating null byte, exceeds size bytes, NULL is returned, and
errno is set to ERANGE; an application should check for this error, and
allocate a larger buffer if necessary.

That's very kind of getcwd to do that, but unfortunately readlink is not so kind.

[chkoreff]

Here's some code that illustrates how to allocate the buffer for readlink dynamically, using lstat:

http://linux.die.net/man/2/readlink

But unfortunately, lstat doesn't return a useable size for entries in the /proc directory, as http://linux.die.net/man/2/stat says:

"For most files under the /proc directory, stat() does not return the file size in the st_size field; instead the field is returned with the value 0. "

[chkoreff]

Which leaves me with this technique for regrowing the buffer:

http://stackoverflow.com/questions/9385386/howto-use-readlink-with-dynamic-memory-allocation

Which is fine, that sounds very solid -- just keep calling readlink until it returns the same size as the previous call. Grow the buffer if needed. I can malloc one buffer of size 1024 or whatever, and call readlink with a lower size than that, then call it again with the full size. If the return values are the same, then I don't have to call malloc a second time. I always have to call readlink at least twice to be sure it returned the full path without truncation, since readlink does not report that error condition explicitly. :(

[chkoreff]

Easier than that -- just check the return length from readlink, and if it's EQUAL to the buffer size, assume the worst and try a larger buffer.

[chkoreff]

OK, I've now added a plug-in so you can call readlink(2) from Fexl:

https://github.com/chkoreff/Fexl/commit/943d69e8f8cb7214d10ba2c4237d818e32d84cfd

I'm not using it yet to eliminate the need for BASE_PATH, but it's a good thing to have in the standard library in any case. If I do use it to emulate $0 in fexl.c, I can just call the inner routine safe_readlink defined in src/file.c.

[chkoreff]

I've now completely eliminated the old base path mechanism involving those stub .c files created in obj. Instead, I now get the base path dynamically using the equivalent of $0 in the /bin/sh program. This is far more elegant, and also good preparation for some other things I mention in the commit message:

https://github.com/chkoreff/Fexl/commit/855ce18a94673e60acea7efda0e2b348cf49b149

Thanks for all your suggestions, falconindy and keenerd -- you've prompted lots of thoughts and opportunities for improvement. To me, the important thing is not the specific tools I do or do not use, such as "make". The important thing is making it behave right.

For example, because I'm still linking the executable to the library, the following doesn't work:

patrick@laptop:~/project/fexl$ ./build install in ../foo
patrick@laptop:~$ project/foo/bin/fexl 
project/foo/bin/fexl: error while loading shared libraries: ../foo/lib/fexl/libfexl.so: cannot open shared object file: No such 
file or directory

Now see I'm being deliberately cruel in my testing there, using a relative path such as ../foo. But it's a good illustration. As I mentioned above, this new dynamic base path feature (which again is the equivalent of /bin/sh's $0), is an excellent first step toward making the ../foo example work correctly, because I'll dlopen the library explicitly, instead of passing 0 to dlsym (in dynamic.c) to use the linked-in library.

It gets even better, because after I do that, I'll be able to move an existing installation to a new place and it will still work, e.g. mv ~/foo ~/tmp/bar, and then run ~/tmp/bar/bin/fexl.

So again, thanks to all.

[chkoreff]

There's a reason I've kept this issue open so long. Lately I've been working on a radically simplified version of Fexl which uses combinators intead of lambda substitution patterns. In short, the function pointers in value structures now have the signature "value type_X(value f, value g)".

In this new version, I've eliminated the shared library altogether. As keenerd suggested, I'm using an explicit look-up table for the built-in functions, instead of relying on dlsym.

There are other radical simplifications as well, and I'm very happy with the results so far. I'll keep you posted when it's ready.

[chkoreff]

Here's the "fresh" branch, vastly simplified: https://github.com/chkoreff/Fexl/tree/fresh

[chkoreff]

Here's an update on my recent progress: http://fexl.com/2012-12-03

This rewrite is going extraordinarily well, and I'm quite chuffed with the new code. I post the announcement here because it's a very roundabout way of addressing the original issues which keenerd and falconindy raised.

[chkoreff]

keenerd wrote:

Trying to package Fexl for Arch Linux. Running into a slew of issues.

You can't build as an unprivileged user.
There is no equivalent to make DESTDIR=...
It does not create /usr/bin or /usr/share if these locations do not already exist.

This is all fixed now, so I am closing the issue.