facebook / zstd

Zstandard - Fast real-time compression algorithm
http://www.zstd.net
Other
23.25k stars 2.07k forks source link

Support read-only source #3741

Open solskogen opened 1 year ago

solskogen commented 1 year ago

If the sources for zstd is located on a read-only filesystem (readonly NFS for instance), even with BUILD_DIR set to a writeable directory, the build will fail:

make: Entering directory '/usr/src/zstd-1.5.5/lib' Installing includes make[1]: Entering directory '/usr/src/zstd-1.5.5/lib' CC /tmp/zstd/dynamic/entropy_common.o CC /tmp/zstd/dynamic/xxhash.o CC /tmp/zstd/dynamic/zstd_compress_sequences.o compiling multi-threaded dynamic library 1.5.5 creating versioned links ln: failed to create symbolic link 'libzstd.so.1': Read-only file system make[1]: [Makefile:163: /tmp/zstd/dynamic/libzstd.so.1.5.5] Error 1 make[1]: Leaving directory '/usr/src/zstd-1.5.5/lib' make: [Makefile:330: install-shared] Error 2 make: Leaving directory '/usr/src/zstd-1.5.5/lib'

Cyan4973 commented 1 year ago

Any default Makefile, using only make default options, would likely fail at the same scenario, since it would generate the *.o object files in the same directories as the source files.

Since this scenario is beyond default, I'm looking for some "industry standard practices" to support read-only source repositories with make, if that ever exists.

In the specific case of the zstd make build script, we have some custom control over where the *.o object files are generated, using the variable BUILD_DIR. However, it only controls the destination of *.o object files, and there are many other files that can be generated, starting with the final binaries themselves, or their symlinks.

That's where the build script fails in your example. These target binaries are still generated in their default directory (lib/ in this case), which are read-only.

A relatively common way to control the destination of generated binaries with Makefile is to employ the variable DESTDIR. However, it still wouldn't work because DESTDIR is for staged installs, meaning that while it controls the final destination of the binaries, there's nothing that states that no other intermediate storage shall ever be used for these binaries. Which is what happens : during the "build" stage, the binaries are generated in their default directories, and in the second "install" stage, they are transferred to their final destination. It mirrors the example provided in the GNU documentation.

Therefore, in order to be able to build from a read-only directory, this intermediate generation stage should be either skipped, or redirected.

That's a non-trivial change, because it's distributed among all targets instead of being concentrated in a single paragraph. Hence it comes with risks, and heavier maintenance.

So, before moving in a direction which would make the project more difficult to maintain, I would like to get an idea of "industry standard practices" if they exist, or consider if other solutions (such as copying the source tree into a writable location) would ultimately prove easier to live with.

solskogen commented 1 year ago

The Makefile for the Linux kernel supports this