Closed probonopd closed 3 years ago
Related question, I would like to search the contents of http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/ which unfortunately gives 403 Forbidden
; and the "Branch" drop-down menu on https://www.freshports.org/search.php does not offer release_0
:
- Query the FreshPorts online database to tell which package/port contains a file with a given name, e.g.,
libIlmImfUtil-2_5.so.24
?
Yes, but it will not get you the results you want.
Look at https://www.freshports.org/graphics/openexr/ - under pkg-plist, click on Expand this list. That is what FreshPorts has.
That might sound like its exactly what you need, but it's not. The database stores only the most recent information for the latest commit.
Thus, you can find libIlmImfUtil-2_5.so.25
but not libIlmImfUtil-2_5.so.24
- For offline use, export a table of the FreshPorts database that contains two columns: 1.) the filename of each file that either ends with
.so
or has.so.
in its name, and 2.) the name of the port this file is contained in. How large would such a table be approximately?
For the pkg-plist
contents, no, there is export like that.
The table contains about 8.5 million lines.
Related question, I would like to search the contents of http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/ which unfortunately gives
403 Forbidden
; and the "Branch" drop-down menu on https://www.freshports.org/search.php does not offerrelease_0
:
What is release_0
? Is that merely head
? If so, that is the default search on FreshPorts.
Thanks @dlangille.
What is release_0? Is that merely head? If so, that is the default search on FreshPorts.
Great question. Now I feel less stupid ;-)
I don't know how exactly http://pkg.freebsd.org/FreeBSD:12:amd64/release_0 relates to what FreshPorts uses.
I don't know what release_0
is. This is the first I've heard of it. It may not be of use to you at all.
Uses for pkg-plist
contents?
FreshPorts uses make generate-plist
- see https://news.freshports.org/2021/03/02/converting-a-column-based-expression-to-a-row-based-solution/
I don't know what
release_0
is.
@igalic said on IRC:
it's a branch (copy) of the quarterly repo at the time of the release as included in the DVD ISO
Ohhh! That's nifty.
So FreshPorts doesn't have data for it because it doesn't keep track of "the time of the release as included in the DVD ISO"?
So FreshPorts doesn't have data for it because it doesn't keep track of "the time of the release as included in the DVD ISO"?
Correct.
FreshPorts update the information for a port when a commit occurs. It always has the latest information for the port. It shows history information in the Commit history
section of the port page, but that does not extend to what pkg-plist
contained at the time of that commit. Only the most recent pkg-plist
information is recorded.
Thanks for the clarification @dlangille.
So it seems like FreshPorts does not have the information I am trying to get.
Say, we need to find libIlmImf-2_3.so.24
.
Here would be my workaround:
graphics/openexr
latest
, quarterly
, release_0
, release_1
, release_2
,...packagesite.yaml
from thereshlibs_provided
for libIlmImf-2_3.so.24
and if found, use repopath
, e.g., All/openexr-2.3.0.txz
to construct http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/openexr-2.3.0.txzDoable, but a lot of downloading just to arrive at the download URL of the package that contains libIlmImf-2_3.so.24
.
Which bears the question:
Wouldn't the information in those packagesite.txz
be goodsomething that FreshPorts would like to learn about?
FreshPorts uses packagesite.txz
to populate the package information:
I don't think the steps outlined will get you want you want. You are still only pulling current information, not historical.
I am basing that on the mention of libIlmImfUtil-2_5.so.24
in the original post.
Let's do this with a practical example:
LD_LIBRARY_PATH
or a comparable method since I don't want to mess up my system with the (older) versions of the dependencies this binary needsrelease_0
, release_1
, release_2
, quarterly
, latest
packagesite.py
release_1
can provide all required dependencies./deps.py ./cgg
# ['libavcodec.so.58', 'libavdevice.so.58', 'libavfilter.so.7', 'libavformat.so.58', 'libavutil.so.56', 'libswresample.so.3', 'libswscale.so.5', 'libintl.so.8', 'libiconv.so.2', 'libtheoraenc.so.1', 'libtheoradec.so.1', 'libogg.so.0', 'libvorbis.so.0', 'libvorbisfile.so.3', 'libIlmImf-2_3.so.24', 'libIlmThread-2_3.so.24', 'libIex-2_3.so.24', 'liba52.so.0', 'libfftw3.so.3', 'libFLAC.so.8', 'libgif.so.7', 'libjpeg.so.8', 'libsndfile.so.1', 'libtheora.so.0', 'libuuid.so.1', 'libvorbisenc.so.2', 'libaom.so.0', 'libdav1d.so.3', 'libwebp.so.7', 'libtiff.so.5', 'libx264.so.157', 'libx265.so.176', 'libvpx.so.6', 'libopus.so.0', 'libX11.so.6', 'libXext.so.6', 'libXinerama.so.1', 'libXfixes.so.3', 'libfontconfig.so.1', 'libfreetype.so.6', 'libpng16.so.16', 'libjbig.so.2', 'libGL.so.1', 'libGLU.so.1', 'libXv.so.1', 'libXft.so.2', 'libpulse-simple.so.0', 'libpulse.so.0']
# http://pkg.freebsd.org/FreeBSD:12:amd64/?C=M&O=A
# Available releases: ['release_0', 'release_1', 'release_2', 'quarterly', 'latest']
# Check if libavcodec.so.58 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/ffmpeg-4.0.3,1.txz
# Check if libavdevice.so.58 is available in 12:amd64/release_0...
# URL already listed above
# Check if libavfilter.so.7 is available in 12:amd64/release_0...
# URL already listed above
# Check if libavformat.so.58 is available in 12:amd64/release_0...
# URL already listed above
# Check if libavutil.so.56 is available in 12:amd64/release_0...
# URL already listed above
# Check if libswresample.so.3 is available in 12:amd64/release_0...
# URL already listed above
# Check if libswscale.so.5 is available in 12:amd64/release_0...
# URL already listed above
# Check if libintl.so.8 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/gettext-runtime-0.19.8.1_1.txz
# Check if libiconv.so.2 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/libiconv-1.14_11.txz
# Check if libtheoraenc.so.1 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/libtheora-1.1.1_7.txz
# Check if libtheoradec.so.1 is available in 12:amd64/release_0...
# URL already listed above
# Check if libogg.so.0 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/libogg-1.3.3,4.txz
# Check if libvorbis.so.0 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/libvorbis-1.3.6,3.txz
# Check if libvorbisfile.so.3 is available in 12:amd64/release_0...
# URL already listed above
# Check if libIlmImf-2_3.so.24 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/openexr-2.3.0.txz
# Check if libIlmThread-2_3.so.24 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/ilmbase-2.3.0.txz
# Check if libIex-2_3.so.24 is available in 12:amd64/release_0...
# URL already listed above
# Check if liba52.so.0 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/liba52-0.7.4_3.txz
# Check if libfftw3.so.3 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/fftw3-3.3.8_1.txz
# Check if libFLAC.so.8 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/flac-1.3.2.txz
# Check if libgif.so.7 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/giflib-5.1.4.txz
# Check if libjpeg.so.8 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/mozjpeg-3.3.1.txz
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/jpeg-turbo-2.0.0.txz
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/jpeg-8_7.txz
# Check if libsndfile.so.1 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/libsndfile-1.0.28_1.txz
# Check if libtheora.so.0 is available in 12:amd64/release_0...
# URL already listed above
# Check if libuuid.so.1 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/e2fsprogs-libuuid-1.44.4.txz
# Check if libvorbisenc.so.2 is available in 12:amd64/release_0...
# URL already listed above
# Check if libaom.so.0 is available in 12:amd64/release_0...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/aom-1.0.0.txz
# Check if libdav1d.so.3 is available in 12:amd64/release_0...
# At least one library was not found in this release, hence trying with the next release
########################################################################################
# Check if libavcodec.so.58 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/ffmpeg-4.2.1_4,1.txz
# Check if libavdevice.so.58 is available in 12:amd64/release_1...
# URL already listed above
# Check if libavfilter.so.7 is available in 12:amd64/release_1...
# URL already listed above
# Check if libavformat.so.58 is available in 12:amd64/release_1...
# URL already listed above
# Check if libavutil.so.56 is available in 12:amd64/release_1...
# URL already listed above
# Check if libswresample.so.3 is available in 12:amd64/release_1...
# URL already listed above
# Check if libswscale.so.5 is available in 12:amd64/release_1...
# URL already listed above
# Check if libintl.so.8 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/gettext-runtime-0.20.1.txz
# Check if libiconv.so.2 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libiconv-1.14_11.txz
# Check if libtheoraenc.so.1 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libtheora-1.1.1_7.txz
# Check if libtheoradec.so.1 is available in 12:amd64/release_1...
# URL already listed above
# Check if libogg.so.0 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libogg-1.3.4,4.txz
# Check if libvorbis.so.0 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libvorbis-1.3.6,3.txz
# Check if libvorbisfile.so.3 is available in 12:amd64/release_1...
# URL already listed above
# Check if libIlmImf-2_3.so.24 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/openexr-2.3.0_2.txz
# Check if libIlmThread-2_3.so.24 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/ilmbase-2.3.0_3.txz
# Check if libIex-2_3.so.24 is available in 12:amd64/release_1...
# URL already listed above
# Check if liba52.so.0 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/liba52-0.7.4_3.txz
# Check if libfftw3.so.3 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/fftw3-3.3.8_4.txz
# Check if libFLAC.so.8 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/flac-1.3.3.txz
# Check if libgif.so.7 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/giflib-5.1.9.txz
# Check if libjpeg.so.8 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/jpeg-turbo-2.0.3.txz
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/jpeg-8_7.txz
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/mozjpeg-3.3.1.txz
# Check if libsndfile.so.1 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libsndfile-1.0.28_2.txz
# Check if libtheora.so.0 is available in 12:amd64/release_1...
# URL already listed above
# Check if libuuid.so.1 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/e2fsprogs-libuuid-1.45.3.txz
# Check if libvorbisenc.so.2 is available in 12:amd64/release_1...
# URL already listed above
# Check if libaom.so.0 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/aom-1.0.0.2439.txz
# Check if libdav1d.so.3 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/dav1d-0.5.0.txz
# Check if libwebp.so.7 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/webp-1.0.3_1.txz
# Check if libtiff.so.5 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/tiff-4.0.10_1.txz
# Check if libx264.so.157 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libx264-0.157.2980.txz
# Check if libx265.so.176 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/x265-3.1.2_2.txz
# Check if libvpx.so.6 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libvpx-1.8.1_1.txz
# Check if libopus.so.0 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/opus-1.3.1.txz
# Check if libX11.so.6 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libX11-1.6.8,1.txz
# Check if libXext.so.6 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libXext-1.3.4,1.txz
# Check if libXinerama.so.1 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libXinerama-1.1.4_2,1.txz
# Check if libXfixes.so.3 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libXfixes-5.0.3_2.txz
# Check if libfontconfig.so.1 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/fontconfig-2.12.6,1.txz
# Check if libfreetype.so.6 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/freetype2-2.10.1.txz
# Check if libpng16.so.16 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/png-1.6.37.txz
# Check if libjbig.so.2 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/jbigkit-2.1_1.txz
# Check if libGL.so.1 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/nvidia-driver-390.87_3.txz
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/nvidia-driver-340-340.107_4.txz
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/nvidia-driver-304-304.137_4.txz
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/mesa-libs-18.3.2_2.txz
# Check if libGLU.so.1 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libGLU-9.0.0_3.txz
# Check if libXv.so.1 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libXv-1.0.11_2,1.txz
# Check if libXft.so.2 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/libXft-2.3.2_3.txz
# Check if libpulse-simple.so.0 is available in 12:amd64/release_1...
http://pkg.freebsd.org/FreeBSD:12:amd64/release_1/All/pulseaudio-13.0.txz
# Check if libpulse.so.0 is available in 12:amd64/release_1...
# URL already listed above
Seems like release_1 can provide all required dependencies
So by downloading the listed URLs (and their dependencies) I should be fine.
Here is the script (quick and dirty but did the job for me):
#!/usr/bin/env python3
import os, sys, socket, subprocess, re, requests, urllib, tarfile, json, fnmatch
if len(sys.argv) <2:
print("USAGE: %s <path to ELF file>" % sys.argv[0])
exit(1)
else:
elf_file = sys.argv[1]
if not os.path.isfile(elf_file):
print ("File does not exist")
exit(1)
# Get the filenames of all dependency libraries using ldd
libs = []
proc = subprocess.Popen(['readelf', '-d', elf_file],stdout=subprocess.PIPE)
i=0
while True:
i = i+1
line = proc.stdout.readline().decode("utf-8").strip()
# print(line)
if not line and i != 1:
# Avoid breaking already on the first line, which is a blank line
break
print(line)
if not "NEEDED" in line:
continue
regex = r".*?NEEDED.*?\[(.*?)\].*?" # https://regex101.com/
matches = re.findall(regex, line, re.MULTILINE)
lib_name = os.path.basename(matches[0])
# print(lib_name)
libs.append(lib_name)
# Determine which libraries are provided by base; those we will not need to search in packages
base_libs_in_lib = fnmatch.filter(os.listdir('/lib'), '*.so*')
base_libs_in_usr_lib = fnmatch.filter(os.listdir('/usr/lib'), '*.so*')
for base_lib_in_lib in base_libs_in_lib:
if base_lib_in_lib in libs:
libs.remove(base_lib_in_lib)
for base_lib_in_usr_lib in base_libs_in_usr_lib:
if base_lib_in_usr_lib in libs:
libs.remove(base_lib_in_usr_lib)
if 'libGL-NVIDIA.so.1' in libs:
libs.remove('libGL-NVIDIA.so.1')
print("# " + str(libs))
# Using http://pkg.freebsd.org/FreeBSD:12:amd64/?C=M&O=A determine latest, quarterly, release_0, release_1, release_2,...
major = "12" # TODO: Get dynamically
cpu_arch = "amd64" # TODO: Get dynamically
baseurl = "http://pkg.freebsd.org/FreeBSD:" + major + ":" + cpu_arch + "/"
url = baseurl + "?C=M&O=A"
print("# " + url)
r = requests.get(url)
# print(r.content.decode("utf-8"))
regex = r"<tr>.*?href=\".*? title=\"(.*?)\">\1/</a>.*?date\">.*?</td></tr>" # https://regex101.com/
releases = re.findall(regex, r.content.decode("utf-8"), re.MULTILINE)
print("# Available releases: " + str(releases)) # Should be in the order of date of release, e.g., ['latest', 'quarterly', 'release_0', 'release_1', 'release_2']
# For each of those, downlad http://pkg.freebsd.org/FreeBSD:12:amd64/RELEASENAME/packagesite.txz and extract packagesite.yaml from there
for release in releases:
path = "/tmp/checkdeps/" + major + ":" + cpu_arch + "/" + release + "/"
# print("# " + path)
os.makedirs(path, exist_ok=True)
url = "http://pkg.freebsd.org/FreeBSD:" + major + ":" + cpu_arch + "/" + release + "/packagesite.txz"
packagesite_file = path + "/packagesite.txz"
packagesite_yaml_file = path + "/packagesite.yaml"
if not os.path.isfile(packagesite_file) and not os.path.isfile(packagesite_yaml_file):
print("Downloading " + packagesite_file)
urllib.request.urlretrieve(url, packagesite_file)
if not os.path.isfile(packagesite_yaml_file):
tar = tarfile.open(packagesite_file, "r:xz")
tar.extract("packagesite.yaml", path=path)
tar.close()
# In each of those, search shlibs_provided for the so file and if found, use repopath,
# e.g., All/openexr-2.3.0.txz to construct http://pkg.freebsd.org/FreeBSD:12:amd64/release_0/All/openexr-2.3.0.txz
urls_to_be_downloaded = []
for release in releases:
found_missing_lib = False
for lib in libs:
found_pkg_for_lib = False
print("# Check if %s is available in %s..." % (lib, major + ":" + cpu_arch + "/" + release))
path = "/tmp/checkdeps/" + major + ":" + cpu_arch + "/" + release + "/"
packagesite_yaml_file = path + "/packagesite.yaml"
with open(packagesite_yaml_file, encoding="ISO-8859-1") as f:
data = [json.loads(line) for line in f]
# print(json.dumps(data, indent=4)) # Hundreds of MBs
for port in data:
if "shlibs_provided" in port.keys():
if lib in port["shlibs_provided"]:
download_url = baseurl + release + "/" + port["path"]
if not download_url in urls_to_be_downloaded:
urls_to_be_downloaded.append(download_url)
print(download_url)
else:
print("# URL already listed above")
found_pkg_for_lib = True
continue
if (found_pkg_for_lib == False):
print("# At least one library was not found in this release, hence trying with the next release")
print("########################################################################################")
found_missing_lib = True
break
if found_missing_lib == False:
print("Seems like %s can provide all required dependencies" % (release))
exit(0)
Feel free to reopen if you think what the script is doing would roughly be interesting for FreshPorts.
Why doesn't FreeBSD's pkg
system automatically install the required dependencies?
https://www.cinelerra-gg.org/freebsd-package/ provides a tar.xz
with the binary of an application., not a package that can be installed with the pkg
command.
Ahh, good, I was afraid you were needing to do this with every package.
They go to all that trouble of creating a binary and don't create a FreeBSD package. This sounds ripe for community support.
In helloSystem we would like to show clear messages in case an application cannot be launched due to missing dependencies, telling the user which package/port to install.
Is there a way to
libIlmImfUtil-2_5.so.24
?.so
or has.so.
in its name, and 2.) the name of the port this file is contained in. How large would such a table be approximately?Debian and Ubuntu allow for similar online searches, e.g., https://packages.debian.org/search?arch=amd64&mode=filename&searchon=contents&keywords=libIlmImf; and the
apt-file
can do it on the local machine.Reference: https://github.com/helloSystem/launch/issues/1#issuecomment-798869996