No Python 3.x-only ports with extra Python dependencies, e.g. "archivers/py-borgbackup"

[liweitianux]

Hello,

I checked the failed packages at http://muscles.dragonflybsd.org/synth/snap-failures/Report/ , and found that archivers/py-borgbackup was ignored due to the requirement of Python >=3.4 (however, the default Python version is 2.7).

However, I cannot find this port in the checked out /usr/dports as well as this DPorts repository. So, how can the above Synth bulk build have such port? And how can I build this port manually or using Synth?

In addition, I manually obtained this port from FreeBSD ports tree, and successfully built (option FUSE=off) and installed it (with Python 3.6) on DFly master. And this software works well.

Cheers, Aly

[jrmarino]

The port is not available. Any port that will not build with the default python (2.7 as you said) is not available. It is a limitation of FreeBSD ports to support exactly 1 version of python simultaneously.

py-borgbackup (from the symptoms) will not build with python-2.7 and in fact requires at least python 3.4. Therefore it was not imported into dports.

[liweitianux]

Hi @jrmarino ,

Thank you for the explanation, and the hard work on DPorts.

However, I'm still confused and have following questions:

1. How can FreeBSD (which I think is also default to Python 2.7) provide the binary package for this port archivers/py-borgbackup as py35-borgbackup[1]?

2. As for the binary packages provided by DFly, I notice that there are ~100 py35-* packages. How does the different Python version been handled for these? Why can't this port archivers/py-borgbackup been similarly dealt with?

3. Maybe Synth by default cannot deal with multiple Python versions simultaneously, can we still import the working ports (e.g., this one) into DPorts, but let user manually build it/make it easier to manually build?

[1] https://www.freshports.org/archivers/py-borgbackup

Cheers, Aly

[jrmarino]

The idea behind dports is that all ports are buildable on the defaults. The only recent exception involves openssl/libressl where we end up with ports that only build under openssl when libressl is the default.

python35-only port could be built with a special profile that defines python default as 3.5, but no binary package would ever be produce because the packaging machine uses the defaults. In general the python strategy is follow the default. It should already be python 3.5 or higher, but for some reason the freebsd python team refuses to change the default version which is EOL in 2.5 years.

To answer your question, I don't know how freebsd makes a binary package unless they do multiple (limited) passes before creating the repository.

Any existing python35 package in dports may exist if there is no recursive dependence on python 2.7. Meaning that if the requires are limited to python35 ONLY, then the port may build. It has to do with how the version of python is determined in freebsd.

As a complete aside, in my new project https://github.com/jrmarino/Ravenports, all versions of python are supported simultaneously and the default version is 3.5. The same issue is not present there.

[jrmarino]

by the way, the py-borgbackup port has a special line in the makefile that prevents the build: https://svnweb.freebsd.org/ports/head/archivers/py-borgbackup/Makefile?revision=434205&view=markup#l31

I don't know why this line is needed. I don't think it's common to do that. Usually the framework takes care of it. Maybe it's a bug? in synth, PACKAGE_BUILDING is always defined.

[liweitianux]

by the way, the py-borgbackup port has a special line in the makefile that prevents the build: https://svnweb.freebsd.org/ports/head/archivers/py-borgbackup/Makefile?revision=434205&view=markup#l31

I don't know why this line is needed. I don't think it's common to do that. Usually the framework takes care of it. Maybe it's a bug? in synth, PACKAGE_BUILDING is always defined.

It seems that this IGNORE directive is a workaround to the bug in Poudriere, which currently cannot handle DEPENDS_ARGS properly [1][2].

However, Synth seems to have similar problem (but I'm not sure). I deleted the IGNORE directive from the port archivers/py-borgbackup, and tried to build it with synth test archivers/py-borgbackup, but failed with:

--------------------------------------------------------------------------------
--  Phase: build-depends
--------------------------------------------------------------------------------
===>   py36-borgbackup-1.0.10 depends on package: py36-setuptools_scm>=1.7 - not found
===>   py36-borgbackup-1.0.10 depends on package: /packages/All/py36-setuptools_scm-1.15.5.txz - not found
===>   USE_PACKAGE_DEPENDS_ONLY set - not building missing dependency from source
*** Error code 1

On the other hand, how can I set Python 3.x (e.g., 3.6) for a Synth profile?

[1] https://lists.freebsd.org/pipermail/freebsd-ports/2016-June/103716.html [2] https://github.com/freebsd/poudriere/issues/259

Cheers, Aly

[jrmarino]

i don't know why bryan is characterizing this as a "poudriere bug". If it's a bug, it's in the framework. Meaning the "forcing dependency on python3" is illegal. If the ports framework is altered to support this, then it would start working in poudriere and synth alike.

as for profiles, see https://github.com/jrmarino/synth#custom-makeconf You would set the default version override there for a specific profile.

[liweitianux]

In the Poudriere issue #259, bryan originally regarded this dependency problem as a Ports Framework bug in 2014 [comment@2014-12-19], but now took it as a bug in Poudriere w.r.t. DEPENDS_ARGS environment variables [comment@2016-02-20]. Therefore, the same problem may occur in Synth. I will close this and open a new issue there to keep track of this problem.

On the other hand, maybe the Ports Framework has too many hacks and is too complicated, which makes a bulk builder (e.g., Synth, Poudriere) cannot Just Work and need take care too many edge cases.

Hope the new Ravenports grow to be a powerful yet beautiful ports collection.

[bdrewery]

" If the ports framework is altered to support this, then it would start working in poudriere and synth alike."

It does though. That was my initial confusion as well. It has supported passing _DEPENDSARGS down the dependency chain "forever". All of the direct "make", portmaster and portupgrade already do the right thing here.

~/svn/ports/net/tiny-network-utilities # make -V RUN_DEPENDS
py36-netifaces>0:net/py-netifaces  py36-hexdump>0:textproc/py-hexdump /usr/local/bin/python3.6:lang/python36

So here we can see that net/tiny-network-utilities wants the py36-netifaces package. This is due to it using USES+= python:34+, but the idea is probably a decade older than USES.

~/svn/ports/net/tiny-network-utilities # make -V DEPENDS_ARGS
PYTHON_VERSION=python3.6

And the py-netifaces port is using the default python version:

~/svn/ports/net/py-netifaces # make -V PKGNAME
py27-netifaces-0.10.5

The way the ports framework handles this is when the run-depends phase builds for net/tiny-network-utilities it will basically run make -C net/py-netifaces DEPENDS_ARGS=PYTHON_VERSION=python3.6 and thus it builds a proper py36-netifaces package. A bug that was in the system for a long time, at least I think, was that you could not then go into net/py-netifaces and install the default 27 version. This is what mva@ was addressing with his uniquefiles changes. Allowing the same origin to be installed multiple times.

Poudriere incorrectly never supported this. I'm not sure what the Tinderbox situation was but it doesn't matter at this point. The solution people came up with was to add a ton of slave ports to work around the problem. It is not maintainable though vs simply listing all of the python versions in a port that the port supports (meaning FLAVORS).

I've added support for _DEPENDSARGS to Poudriere at https://github.com/freebsd/poudriere/pull/483 and will merge it in the next few days. Next I will add FLAVORS support which explicitly lists multiple packages that a port can spit out. The port should be built with make FLAVOR=foo to generate the proper package. The major work for supporting _DEPENDSARGS in Poudriere was mostly to support multiple packages from the same origin. Adding support for that basically gives _DEPENDSARGS support and FLAVORS support for free.

I'm not familiar with the Synth codebase, but the major item that needs to be done would be to change the queue to be PKGNAME based and to generate items to the queue based on ORIGIN + OTHER STUFF, OTHER STUFF being that DEPENDS_ARGS value and FLAVORS value. For port A with DEPENDS_ARGS being FOO with deps B C D, when the dependencies and PKGNAME etc are looked up for B C D you need to pass make DEPENDS_ARGS=FOO -V PKGNAME ... as a make argument not environment. Those B C, and D ports will later be processed again with either blank or the same _DEPENDSARGS from another port. It may generate a new unique PKGNAME. It may also generate a superfluous PKGNAME that can be dropped.

For the superfluous case consider net/tiny-network-utilities wanting a dependency on ports-mgmt/pkg. It will pass DEPENDS_ARGS=PYTHON_VERSION=python34 to the metadata lookup for ports-mgmt/pkg and reveal that the PKGNAME is the same as not passing it at all. So the excess lookup can be dropped and not enqueued.

Finally I'm not quite sure how FLAVORS and _DEPENDSARGS will interact, but each are separate problems to deal with. FLAVORS will likely take months-years to work out completely, all while _DEPENDSARGS will continue to be supported.

[bdrewery]

Having said all of that, just forget about _DEPENDSARGS. Only FLAVORS/FLAVOR will need to be implemented. We will be removing _DEPENDSARGS hacks entirely in lieu of FLAVOR (which will work very similarly to _DEPENDSARGS). A mail will be going to ports@ soon about what exactly is needed to support it.