[CFEP-19] pinning epochs

[beckermr]

Hi @conda-forge/core!

After a bunch of the feedback from last time, I have posted a new version of the pinning epochs CFEP for comments.

[beckermr]

Thanks @isuruf!

1. I don't know and I don't actually think this question is relevant to this discussion. We as conda-forge get to decide our own policies. It doesn't matter if other ppl do or don't do this. Maybe they should?

2. Nothing in the cfep prevents us from deprecating builds against a given epoch early. The requirements are that we have at most 2 and don't tag them more often than every six months.

3. I've addressed this question in the cfep above and in person. It benefits anyone wanting longer term abi stability than we currently provide. The premise that this is only corporations is not true and misguided. The group I am working with personally is a group of scientists. This particular group has already contributed to conda-forge and even offered to help with the cos7 stuff. The current system would work like aarch or ppc64le and so is consistent from a policy point of view.

[isuruf]

I don't know and I don't actually think this question is relevant to this discussion. We as conda-forge get to decide our own policies. It doesn't matter if other ppl do or don't do this. Maybe they should?

No, it would be stupid of us not to look at what others does. No package manager doing this is a sign that this is not scalable.

Nothing in the cfep prevents us from deprecating builds against a given epoch early. The requirements are that we have at most 2 and don't tag them more often than every six months.

Then, this proposal is useless. Even if there's one package that needs to be removed, we'll have to remove all previous epochs. If not, we'll have to keep supporting a package well past its end of life just because there's an epoch that somebody needs. For eg: just because somebody needs boost-cpp, we'll have to keep supporting python X when there's no relationship between them.

The current system would work like aarch or ppc64le and so is consistent from a policy point of view.

That is only for a smaller set of packages, but here we will be doing unnecessary rebuilds for a large amount. For eg: let's say we wanted to have an older pinning with boost-cpp. Then boost (python bindings) need to be rebuilt. Then we'd look at the dependencies and see that there's numpy, so then numpy needs a build with the old epoch which is totally unnecessary.

[isuruf]

The requirements are that we have at most 2 and don't tag them more often than every six months.

So, this is first come first serve basis? Say one person wants the pinning from 3 months ago, but they can't propose it because another person has tagged a pinning less than 6 months ago?

[beckermr]

A few comments.

I don't personally take what other package managers do or don't do as a sign of scalability.

For deprecation, this cfep actually provides a very nice mechanism six months before a major deprecation we can remove the deprecated version from the latest pinnings and only support it in a pinning epoch. Then we let that pinning epoch run itself out. We could do that even earlier if we wanted.

Yes I envision a first come, first serve model for declaring pinnings.

For the specific case of numpy, it is in the pinnings file itself and so is not subject to this cfep. In other words, we are allowing ppl to get a build of any feedstock not in the pinnings for a small subset of the old pinning files. This specific restriction is important since it eliminates these kinds of extra builds of hard / big things in the pinnings file.

[isuruf]

For deprecation, this cfep actually provides a very nice mechanism six months before a major deprecation we can remove the deprecated version from the latest pinnings and only support it in a pinning epoch. Then we let that pinning epoch run itself out. We could do that even earlier if we wanted.

Since, this will be opt-in, then those who didn't opt in will find that they are suddenly losing a build because they are in the latest pinning only.

[scopatz]

Thanks for putting this in @beckermr! I would love to have this include a discussion of the epoch naming scheme (e.g. 2020.06) just so this is clear and there is no debate about it.

[isuruf]

For the specific case of numpy, it is in the pinnings file itself and so is not subject to this cfep. In other words, we are allowing ppl to get a build of any feedstock not in the pinnings for a small subset of the old pinning files. This specific restriction is important since it eliminates these kinds of extra builds of hard / big things in the pinnings file.

Say, we have a package that depends on both scipy and boost-cpp, then the same argument applies and there's a needless rebuild of scipy

[beckermr]

Since, this will be opt-in, then those who didn't opt in will find that they are suddenly losing a build because they are in the latest pinning only.

Yes, this is exactly what would happen and is exactly what we did with our latest big deprecation, python 2.7. We turned it off at some point and then provided people an admin command to add it back to the feedstock if they wanted it.

Say, we have a package that depends on both scipy and boost-cpp, then the same argument applies and there's a needless rebuild of scipy

According to the scheme above, the only time a package is rebuilt is if when rendering, conda-smithy generates a new config file in .ci_support that is different than any of the ones in the feedstock currently. Whether or not this would happen in a specific case depends on the various pinning files involved. If the relevant pins for scipy have not moved, then the feedstock would not change upon rerendering and thus no rebuild would be needed.

[beckermr]

I agree @scopatz. I have added a year month scheme.

[beckermr]

@isuruf I've added the following to the spec to help eliminate a few extra rebuilds of purely >=9-like stuff:

 - Certain, forward-compatible pins corresponding to constraints like `>=9` for compiler runtimes can be copied 
   from the newer pinning file into the pinning epoch in order to eliminate potential rebuilds.

[isuruf]

Yes, this is exactly what would happen and is exactly what we did with our latest big deprecation, python 2.7.

No, we supported python 2.7 until its end of life. What you're suggesting is to proactivaely remove it 6-12 months before reaching its end of life.

[beckermr]

No this is not right. Remember that the potential scheme I suggested is official support too. Should we accept this CFEP, the pinning epochs would be a sanctioned part of conda-forge with the support that entails. Removing a build of a release reaching end of life from latest and onto a pinning epoch doesn't mean we don't support it. It means that to use the older build, one would have to enable it. This is the same as supporting aarch or ppc64le or anything else we have really.

In any case, I want to focus the discussion on a way forward with this CFEP.

[isuruf]

Removing a build of a release reaching end of life from latest and onto a pinning epoch doesn't mean we don't support it. It means that to use the older build, one would have to enable it.

Why? This way you are forcing everybody into this scheme and besides it's only with older pinning and not newer pinning either.

This is the same as supporting aarch or ppc64le or anything else we have really.

No, it's not. We are not dropping aarch or ppc64le.

In any case, I want to focus the discussion on a way forward with this CFEP.

I don't understand. What do you want to do?

[beckermr]

I think we are splitting hairs on what support means in the context of conda forge. This specific deprecation scheme is not a part or even required by this cfep.

What I mean by a way forward is that (I am pretty sure) we have users who have a need for more ABI stability. I want to find a way to meet this need. If it is not this CFEP in its current form, that is totally fine by me. However, I'd like to continue working toward providing this. Instead right now we seem to be caught up in fine details and not focusing on the bigger picture.

[xhochy]

I'm a bit confused by the mentioned "unnecessary rebuilds". From what I have understood the PE proposal is that we would build new versions of a feedstock with an old set of pinnings. Thus we would expand the build matrix in the packages that are not part of the pinnings file but wouldn't touch the builds of the pinned packages.

[beckermr]

I'm a bit confused by the mentioned "unnecessary rebuilds". From what I have understood the PE proposal is that we would build new versions of a feedstock with an old set of pinnings. Thus we would expand the build matrix in the packages that are not part of the pinnings file but wouldn't touch the builds of the pinned packages.

I agree @xhochy. Most of the rebuilds seem needed. There is an edge case for compilers where if everything else is the same, we can use a build with say clang 9 at runtime with clang 10. We can adjust the pinnings file to prevent these which i've added to the spec.

[pkgw]

So, my read of the text is that this CFEP would basically start conda-forge having Long-Term Support (LTS) release branches. (The term does show up once in the CFEP text.) On the one hand, I think that the general Linux distro experience has shown that having an LTS branch can be very valuable and useful for a lot of people. On the other hand, it would probably be a nontrivial amount of effort to support — it's right there in the name.

Perhaps we should try to acquire some more data about how useful this feature would be to our userbase? I'm not a fan of decision-by-survey, but it's also not obvious to me how else to gather information — perhaps look at download trends for older versions of common core packages, to try to get a sense as to how many people are locking on to older versions for stability?

As another thought, would it be at all feasible to somehow prototype this concept outside of mainline conda-forge, and "merge it in" if successful? I'm pretty sure that the answer is "no", since so much depends on the build and update infrastructure. But in general, I think this concept is going to have a lot of uncertainty about both the costs and the benefits, and in those circumstances you want to prototype or otherwise start small. Perhaps one path forward would be to give the idea a trial run, maintaining an LTS branch for a year or so, and evaluate and make a decision about whether we feel that it's worthwhile to continue support or not.

[beckermr]

Agreed @pkgw that we don't have a ton of good info on who might use this. I know of at least one group personally. @xhochy mentioned the arrow folks might as well. OTOH, this has not stopped us from adding features or making changes in the past. So I don't think we should condition this specific issue on a survey or the like.

The original draft of this CFEP used the term "long-term support" in it. We moved away from the term "long-term support" because sometimes people connotate that with backports of critical bug fixes and the like. We are not proposing to do any backports of that nature here, and so we went with the term "pinning epoch" instead.

A prototype is ~impossible to standup given how things are now. :/

[isuruf]

I'm a bit confused by the mentioned "unnecessary rebuilds". From what I have understood the PE proposal is that we would build new versions of a feedstock with an old set of pinnings. Thus we would expand the build matrix in the packages that are not part of the pinnings file but wouldn't touch the builds of the pinned packages.

See my example with scipy. We'll be re-building it just because the compiler version is bumped, but there's no reason to do it. Let's also mention that the only use-case we have right now is for boost and this proposal is taking it to another level.

[beckermr]

See my example with scipy. We'll be re-building it just because the compiler version is bumped, but there's no reason to do it.

@isuruf See my change to the proposal and the comment I left here: https://github.com/conda-forge/cfep/pull/35#issuecomment-664078424

I've eliminated this issue.

[beckermr]

Here is an example of a subset of a diff one gets for scipy against a pinning file from January.

diff --git a/.ci_support/linux_ppc64le_numpy1.16python3.8.____cpython.yaml b/.ci_support/linux_ppc64le_python3.8.____cpython.yaml
similarity index 93%
rename from .ci_support/linux_ppc64le_numpy1.16python3.8.____cpython.yaml
rename to .ci_support/linux_ppc64le_python3.8.____cpython.yaml
index 0213c17..9c7c0ea 100644
--- a/.ci_support/linux_ppc64le_numpy1.16python3.8.____cpython.yaml
+++ b/.ci_support/linux_ppc64le_python3.8.____cpython.yaml
@@ -30,6 +30,3 @@ pin_run_as_build:
     max_pin: x.x
 python:
 - 3.8.* *_cpython
-zip_keys:
-- - numpy
-  - python
diff --git a/.ci_support/linux_numpy1.16python3.6.____cpython.yaml b/.ci_support/linux_python3.6.____cpython.yaml
similarity index 91%
rename from .ci_support/linux_numpy1.16python3.6.____cpython.yaml
rename to .ci_support/linux_python3.6.____cpython.yaml
index c77e7e3..17a66df 100644
--- a/.ci_support/linux_numpy1.16python3.6.____cpython.yaml
+++ b/.ci_support/linux_python3.6.____cpython.yaml
@@ -23,13 +23,10 @@ libcblas:
 liblapack:
 - 3.8.0 *netlib
 numpy:
-- '1.16'
+- '1.14'
 pin_run_as_build:
   python:
     min_pin: x.x
     max_pin: x.x
 python:
 - 3.6.* *_cpython
-zip_keys:
-- - numpy
-  - python

There are two features here:

1. The zip_keys block that was removed is from the pypy migration yaml. This is a special case and we could potentially code around it in smithy by having smithy remove any excess zip keys blocks in the final configs.

2. The minimum version of numpy for building has changed. This might fall under our forward-compatible pins can be updated rule, which would eliminate that build.

Thus I don't think the proposal as stated would generate rebuilds if we are careful.

Here is the pinnings file. I adjusted the pinning file to use the current compiler runtimes and channels per the CFEP.

```yaml # This differs from target_platform in that it determines what subdir the compiler # will target, not what subdir the compiler package will be itself. # For example, we need a win-64 vs2008_win-32 package, so that we compile win-32 # code on win-64 miniconda. cross_compiler_target_platform: # [win] - win-64 # [win] c_compiler: - gcc # [linux] - clang # [osx] - vs2017 # [win] c_compiler_version: # [unix] - 10 # [osx] - 7 # [linux64 or aarch64] - 8 # [ppc64le or armv7l] cxx_compiler: - gxx # [linux] - clangxx # [osx] - vs2017 # [win] cxx_compiler_version: # [unix] - 10 # [osx] - 7 # [linux64 or aarch64] - 8 # [ppc64le or armv7l] fortran_compiler: # [unix or win64] - gfortran # [(linux64 or osx)] - gfortran # [aarch64 or ppc64le or armv7l] - flang # [win64] fortran_compiler_version: # [unix or win64] - 7 # [linux64 or osx or aarch64] - 8 # [ppc64le or armv7l] - 5 # [win64] m2w64_c_compiler: # [win] - m2w64-toolchain # [win] m2w64_cxx_compiler: # [win] - m2w64-toolchain # [win] m2w64_fortran_compiler: # [win] - m2w64-toolchain # [win] CMAKE_GENERATOR: # [win] - NMake Makefiles # [win] cuda_compiler: # [linux64] - nvcc # [linux64] cuda_compiler_version: # [linux64] - None # [linux64] - 9.2 # [linux64] - 10.0 # [linux64] - 10.1 # [linux64] - 10.2 # [linux64] _libgcc_mutex: - 0.1 conda_forge # # Go Compiler Options # # The basic go-compiler with CGO disabled, # It generates fat binaries without libc dependencies # The activation scripts will set your CC,CXX and related flags # to invalid values. go_compiler: - go-nocgo # The go compiler build with CGO enabled. # It can generate fat binaries that depend on conda's libc. # You should use this compiler if the underlying # program needs to link against other C libraries, in which # case make sure to add 'c,cpp,fortran_compiler' for unix # and the m2w64 equivalent for windows. cgo_compiler: - go-cgo # The following are helpful variables to simplify go meta.yaml files. target_goos: - linux # [linux] - darwin # [osx] - windows # [win] target_goarch: - amd64 # [x86_64] target_goexe: - # [unix] - .exe # [win] target_gobin: - '${PREFIX}/bin/' # [unix] - '%PREFIX%\bin\' # [win] # Rust Compiler Options rust_compiler: - rust - rust-gnu # [win] rust_compiler_version: - 1.40.0 macos_min_version: # [osx] - 10.9 # [osx] macos_machine: # [osx] - x86_64-apple-darwin13.4.0 # [osx] MACOSX_DEPLOYMENT_TARGET: # [osx] - 10.9 # [osx] target_platform: # [win] - win-64 # [win] VERBOSE_AT: - V=1 VERBOSE_CM: - VERBOSE=1 # dual build configuration channel_sources: - conda-forge,defaults # [not (aarch64 or armv7l)] - conda-forge # [aarch64] - conda-forge,c4armv7l,defaults # [armv7l] channel_targets: - conda-forge main docker_image: # [linux] - condaforge/linux-anvil-comp7 # [linux64] - condaforge/linux-anvil-cuda:9.2 # [linux64] - condaforge/linux-anvil-cuda:10.0 # [linux64] - condaforge/linux-anvil-cuda:10.1 # [linux64] - condaforge/linux-anvil-cuda:10.2 # [linux64] - condaforge/linux-anvil-aarch64 # [aarch64] - condaforge/linux-anvil-ppc64le # [ppc64le] - condaforge/linux-anvil-armv7l # [armv7l] zip_keys: - - qt - - python - # [linux64] - cuda_compiler_version # [linux64] - docker_image # [linux64] # aarch64 specifics because conda-build sets many things to centos 6 # this can probably be removed when conda-build gets updated defaults # for aarch64 cdt_arch: aarch64 # [aarch64] cdt_name: cos7 # [aarch64] BUILD: aarch64-conda_cos7-linux-gnu # [aarch64] # armv7l specifics because conda-build sets many things to centos 6 # this can probably be removed when conda-build gets updated defaults # for aarch64 cdt_arch: armv7l # [armv7l] cdt_name: cos7 # [armv7l] BUILD: armv7-conda_cos7-linux-gnueabihf # [armv7l] # TODO: remove these when run_exports are added to the packages. pin_run_as_build: arpack: max_pin: x.x.x boost: max_pin: x.x.x boost-cpp: max_pin: x.x.x bzip2: max_pin: x cairo: max_pin: x.x cudnn: max_pin: x min_pin: x.x curl: max_pin: x dbus: max_pin: x expat: max_pin: x.x ffmpeg: max_pin: x.x fftw: max_pin: x flann: max_pin: x.x.x fontconfig: max_pin: x freetype: max_pin: x gdal: max_pin: x.x gdk_pixbuf: max_pin: x.x.x geotiff: max_pin: x.x.x glew: max_pin: x.x glpk: max_pin: x.x gmp: max_pin: x graphviz: max_pin: x grpc-cpp: max_pin: x.x harfbuzz: max_pin: x hdf4: max_pin: x.x icu: max_pin: x isl: max_pin: x.x jasper: max_pin: x jpeg: max_pin: x libjpeg_turbo: max_pin: x json-c: max_pin: x.x jsoncpp: max_pin: x.x.x kealib: max_pin: x.x krb5: max_pin: x.x libblitz: max_pin: x.x libcurl: max_pin: x libevent: max_pin: x.x.x libffi: max_pin: x.x libgdal: max_pin: x.x libiconv: max_pin: x.x libkml: max_pin: x.x libmatio: max_pin: x.x libpcap: max_pin: x.x libpng: max_pin: x.x librdkafka: max_pin: x.x.x librsvg: max_pin: x libssh2: max_pin: x.x libsvm: max_pin: x.x libunwind: max_pin: x.x libtiff: max_pin: x libwebp: max_pin: x.x libxml2: max_pin: x.x libuuid: max_pin: x lz4-c: max_pin: x.x.x lzo: max_pin: x metis: max_pin: x.x mkl: max_pin: x mpfr: max_pin: x ncurses: max_pin: x.x netcdf-cxx4: max_pin: x.x netcdf-fortran: max_pin: x.x nettle: max_pin: x.x nlopt: max_pin: x.x.x nss: max_pin: x nspr: max_pin: x occt: max_pin: x.x openblas: max_pin: x.x.x openturns: max_pin: x.x openjpeg: max_pin: x.x openssl: max_pin: x.x.x pango: max_pin: x.x perl: max_pin: x.x.x pixman: max_pin: x.x poppler: max_pin: x.x qt: max_pin: x.x readline: max_pin: x r-base: max_pin: x.x min_pin: x.x snappy: max_pin: x.x.x sox: max_pin: x.x.x sqlite: max_pin: x sundials: max_pin: x.x tk: max_pin: x.x tiledb: max_pin: x.x.x vc: max_pin: x vlfeat: max_pin: x.x.x vtk: max_pin: x.x.x xerces-c: max_pin: x.x.x xz: max_pin: x.x zeromq: max_pin: x.x # [not win] max_pin: x.x.x # [win] zlib: max_pin: x.x zstd: max_pin: x.x.x # Pinning packages # blas libblas: - 3.8 *netlib libcblas: - 3.8 *netlib liblapack: - 3.8.0 *netlib liblapacke: - 3.8.0 *netlib blas_impl: - openblas - mkl # [x86 or x86_64] - blis # [x86 or x86_64] arpack: - 3.6.3 arrow_cpp: - 0.15.1 boost: - 1.70.0 boost_cpp: - 1.70.0 bzip2: - 1 cairo: - 1.16 cfitsio: - 3.470 cudnn: - 7.6.5 curl: - 7.64 dbus: - 1 expat: - 2.2 ffmpeg: - 4.1 fftw: - 3 flann: - 1.9.1 fontconfig: - 2.13 freetype: - 2.9.1 gf2x: - 1.2 gdk_pixbuf: - 2.36.12 gsl: - 2.5 gstreamer: - 1.14.4 gst_plugins_base: - 1.14.4 gdal: - 3.0 geos: - 3.8.0 geotiff: - 1.5.1 giflib: - 5.1.7 glew: - 2.0.0 glib: - 2.58 glog: - 0.4.0 glpk: - 4.65 gmp: - 6 graphviz: - 2.40 grpc_cpp: - 1.25 harfbuzz: - 2 hdf4: - 4.2 hdf5: - 1.10.5 icu: - 64.2 isl: - 0.19 jasper: - 1.900.1 jpeg: - 9 libjpeg_turbo: - 2 json_c: - 0.13 jsoncpp: - 1.8.4 kealib: - 1.4.10 krb5: - 1.16 libarchive: - 3.3 libblitz: - 0.10 libcurl: - 7.64 libdap4: - 3.20.2 libevent: - 2.1.10 libffi: - 3.2 libgdal: - 3.0 libiconv: - 1.15 libkml: - 1.3 libmatio: - 1.5 libnetcdf: - 4.7.3 libpcap: - 1.8 libpng: - 1.6 libprotobuf: - 3.10 librdkafka: - 0.11.5 librsvg: - 2 libsecret: - 0.18 libspatialindex: - 1.9.3 libssh2: - 1.8 libsvm: - 3.21 libtiff: - 4.1.0 libunwind: - 1.2 libwebp: - 1.0 libxml2: - 2.9 libuuid: - 2.32.1 lz4_c: - 1.8.3 lzo: - 2 metis: - 5.1 mkl: - 2019 mpich: - 3.3 mpfr: - 4 mumps_mpi: - 5.2 mumps_seq: - 5.2 nccl: - 2.4.6.1 ncurses: - 6.1 netcdf_cxx4: - 4.3 netcdf_fortran: - 4.5 nettle: - 3.4 nss: - 3.39 nspr: - 4.20 nlopt: - 2.6.* ntl: - 11.3.2 # we build for the oldest version possible of numpy for forward compatibility numpy: - 1.14 # [not (aarch64 or ppc64le)] - 1.16 # [aarch64 or ppc64le] occt: - 7.4 openblas: - 0.3.6 openjpeg: - 2.3 openmpi: - 4.0 openssl: - 1.1.1a openturns: - 1.14* pango: - 1.42 pari: - 2.11 perl: - 5.26.2 petsc: - 3.12 petsc4py: - 3.12 slepc: - 3.12 slepc4py: - 3.12 pixman: - 0.38 poppler: - 0.67.0 proj: - 6.3.0 python: - 3.6.* *_cpython - 3.7.* *_cpython - 3.8.* *_cpython python_impl: - cpython qt: - 5.12 readline: - 8.0 root: - 6.18.04 root_base: - 6.18.04 root_binaries: - 6.18.04 ruby: - 2.5 - 2.6 r_base: - 3.5.1 - 3.6 scotch: - 6.0.8 ptscotch: - 6.0.8 singular: - 4.1.2 snappy: - 1.1.7 sox: - 14.4.2 sqlite: - 3 sundials: - 3.2 suitesparse: - 5.6 tk: - 8.6 # [not ppc64le] tiledb: - 1.7.0 vc: # [win] - 14 # [win] vlfeat: - 0.9.20 vtk: - 8.2.0 x264: - 1!152.* xerces_c: - 3.2.2 xrootd: - 4.11.0 xz: - 5.2 zeromq: - 4.3.2 zlib: - 1.2 zstd: - 1.4.4 ```

[beckermr]

Also, if we kept the numpy min version, that is 3 more builds than we had, which is pretty small.

[beckermr]

I've adjusted the text again to make sure we state that we can/will adjust the pinnings to avoid extra rebuilds including cases like big deprecations.

[beckermr]

@conda-forge/core Are we ok to merge this as deferred?