timvideos / litex-buildenv

An environment for building LiteX based FPGA designs. Makes it easy to get everything you need!
BSD 2-Clause "Simplified" License
212 stars 79 forks source link

conda: latest gcc-${CPU}-elf-newlib conflicts with latest conda binutils, etc #123

Closed ewenmcneill closed 5 years ago

ewenmcneill commented 5 years ago

We use gcc-${CPU}-elf-newlib to build MicroPython (eg, scripts/build-micropython.sh will install it).

It appears that the most recent conda updates for the main gcc, binutils, etc, conflict with the previous conda builds of gcc-${CPU}-elf-newlib. It looks like this got broken late 2018 (201812xx).

This means it's possible to:

  1. upgrade everything except gcc-${CPU}-elf-newlib, which results in gcc-${CPU}-elf-newlib being downgraded to 5.4.0 (from 8.2.x); or

  2. upgrade gcc-${CPU}-elf-newlib, which results in the main gcc, bintuils, flterm, etc, being downgraded to a 2018-11-xx build (from a 2018-12-xx) build.

My guess is that maybe:

need a conda rebuild for the new binutils so as to avoid the conflicts.

Is it simple to get conda to rebuild gcc-lm32-elf-newlib and gcc-or1k-elf-newlib to match the latest dependencies?

Example output of conda upgrade attempts, and the resulting package upgrades/downgrades below. The same sort of downgrades of gcc-${CPU}-elf-newlib happens when running scripts/download-env.sh again to update the packages (eg, flterm), but that output is longer so I didn't capture it.

(Of note swapping back and forth between gcc 8.2.0 and gcc 5.4.0 will lead to header include issues, due to path caching, unless you do something like make clean in between.)

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda upgrade gcc-lm32-elf-newlib
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /src/fpga/litex-buildenv/build/conda

  added / updated specs:
    - gcc-lm32-elf-newlib

The following packages will be UPDATED:

  gcc-lm32-elf-newl~                  5.4.0-20180626_181010 --> 8.2.0-20181223_002105

The following packages will be DOWNGRADED:

  binutils-lm32-elf                  2.31.1-20181223_002104 --> 2.31.1-20181215_231016
  binutils-or1k-elf                  2.31.1-20190223_232653 --> 2.31.1-20181223_002044
  binutils_linux-64                       2.31.1-h6176602_6 --> 2.31.1-h6176602_3
  flterm                  2.4_0029_g47d3b15-20190224_000016 --> 2.4_0024_gee93960-20181223_010808
  gcc-lm32-elf-nost~                  8.2.0-20190223_232653 --> 8.2.0-20181215_231127
  gcc-or1k-elf-nost~ 8.2.0_3006_g592479378c3-20190223_2331~ --> 8.2.0_3006_g592479378c3-20181223_002632
  gcc_linux-64                             7.3.0-h553295d_6 --> 7.3.0-h553295d_3
  gxx_linux-64                             7.3.0-h553295d_6 --> 7.3.0-h553295d_3

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda upgrade gcc-or1k-elf-newlib
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /src/fpga/litex-buildenv/build/conda

  added / updated specs:
    - gcc-or1k-elf-newlib

The following packages will be UPDATED:

  gcc-or1k-elf-newl~ 5.4.0_4334_g9310fdc97ee-20180626_1811~ --> 8.2.0_3006_g592479378c3-20181215_232223

The following packages will be DOWNGRADED:

  binutils-or1k-elf                  2.31.1-20181223_002044 --> 2.31.1-20181126_032711
  gcc-or1k-elf-nost~ 8.2.0_3006_g592479378c3-20181223_0026~ --> 8.2.0_3006_g592479378c3-20181126_033222

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda upgrade flterm
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /src/fpga/litex-buildenv/build/conda

  added / updated specs:
    - flterm

The following packages will be UPDATED:

  binutils-lm32-elf                  2.31.1-20181215_231016 --> 2.31.1-20181223_002104
  binutils-or1k-elf                  2.31.1-20181126_032711 --> 2.31.1-20190223_232653
  binutils_linux-64                       2.31.1-h6176602_3 --> 2.31.1-h6176602_6
  flterm                  2.4_0024_gee93960-20181223_010808 --> 2.4_0029_g47d3b15-20190224_000016
  gcc-lm32-elf-nost~                  8.2.0-20181215_231127 --> 8.2.0-20190223_232653
  gcc-or1k-elf-nost~ 8.2.0_3006_g592479378c3-20181126_0332~ --> 8.2.0_3006_g592479378c3-20190223_233103
  gcc_linux-64                             7.3.0-h553295d_3 --> 7.3.0-h553295d_6
  gxx_linux-64                             7.3.0-h553295d_3 --> 7.3.0-h553295d_6

The following packages will be DOWNGRADED:

  gcc-lm32-elf-newl~                  8.2.0-20181223_002105 --> 5.4.0-20180626_181010
  gcc-or1k-elf-newl~ 8.2.0_3006_g592479378c3-20181215_2322~ --> 5.4.0_4334_g9310fdc97ee-20180626_181158

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 
ewenmcneill commented 5 years ago

If one attempts to leave the downgraded gcc-${CPU}-elf-newlib packages installed and build MicroPython, then the compiler is unable to find cc1, apparently due to the 8.2.0 cc1 being installed (in a path specific to that version), but the downgraded packages are gcc 5.4.0, a dramatically different version.

It looks like the conda gcc-${CPU}-elf-newlib package does not include its own cc1, and has been relying on the "main" compilers cc1, which only works if the gcc versions are very similar.

Due to the dependency issues, it's not possible to have the latest flterm and the latest gcc-${CPU}-elf-newlib installed at the same time :-(

Thus it seems like currently one needs to do:

conda upgrade gcc-${CPU}-elf-newlib
scripts/build-micropython.sh
conda upgrade flterm
make firmware-connect

to (a) be able to build MicroPython via litex-buildenv and (b) use the latest version of flterm with the "exit cleanly" feature.

Ewen

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ scripts/build-micropython.sh 
              Platform: mimasv2
                Target: base (default: base)
                   CPU: lm32 (default: lm32)
               Firmare: micropython (default: firmware)
+ set -e
+ '[' micropython '!=' micropython ']'
+ lm32-elf-newlib-gcc --version
+ MPY_SRC_DIR=/src/fpga/litex-buildenv/third_party/micropython
+ '[' '!' -d /src/fpga/litex-buildenv/third_party/micropython ']'
+ '[' '!' -d build/mimasv2_base_lm32//software/include/generated ']'
+ LITEX_INCLUDE_BASE=/src/fpga/litex-buildenv/third_party/litex/litex/soc/software/include/base
+ for FILE in system.h csr-defs.h spr-defs.h
+ cp -p /src/fpga/litex-buildenv/third_party/litex/litex/soc/software/include/base/system.h build/mimasv2_base_lm32//software/include
+ for FILE in system.h csr-defs.h spr-defs.h
+ cp -p /src/fpga/litex-buildenv/third_party/litex/litex/soc/software/include/base/csr-defs.h build/mimasv2_base_lm32//software/include
+ for FILE in system.h csr-defs.h spr-defs.h
+ cp -p /src/fpga/litex-buildenv/third_party/litex/litex/soc/software/include/base/spr-defs.h build/mimasv2_base_lm32//software/include
+ TARGET_MPY_BUILD_DIR=build/mimasv2_base_lm32//software/micropython
+ '[' '!' -e build/mimasv2_base_lm32//software/micropython/generated ']'
++ realpath build/mimasv2_base_lm32//software/micropython
+ TARGET_MPY_BUILD_DIR=/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython
+ export CROSS_COMPILE=lm32-elf-newlib-
+ CROSS_COMPILE=lm32-elf-newlib-
++ realpath build/mimasv2_base_lm32//software/include
+ export BUILDINC_DIRECTORY=/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/include
+ BUILDINC_DIRECTORY=/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/include
++ realpath /src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython
+ export BUILD=/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython
+ BUILD=/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython
+ OLD_DIR=/src/fpga/litex-buildenv
+ cd /src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython
++ realpath ../../../../third_party/micropython/ports/fupy/
+ make V=1 -C /src/fpga/litex-buildenv/third_party/micropython/ports/fupy -j4
make: Entering directory '/src/fpga/litex-buildenv/third_party/micropython/ports/fupy'
python ../../py/makeversionhdr.py /src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython/genhdr/mpversion.h
GEN /src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython/genhdr/qstr.i.last
lm32-elf-newlib-gcc -E -DNO_QSTR -I/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython/tmp -mbarrel-shift-enabled -mmultiply-enabled -mdivide-enabled -msign-extend-enabled -I. -I../.. -I../../lib/mp-readline -I/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython -I/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/include -Wall -Werror -std=gnu11 -ggdb  -Og -Wdouble-promotion -Wall -Werror -DNDEBUG ../../py/mpstate.c ../../py/malloc.c ../../py/gc.c ../../py/pystack.c ../../py/qstr.c ../../py/vstr.c ../../py/mpprint.c ../../py/unicode.c ../../py/mpz.c ../../py/reader.c ../../py/lexer.c ../../py/parse.c ../../py/scope.c ../../py/compile.c ../../py/emitcommon.c ../../py/emitbc.c ../../py/asmbase.c ../../py/asmx64.c ../../py/emitnx64.c ../../py/asmx86.c ../../py/emitnx86.c ../../py/asmthumb.c ../../py/emitnthumb.c ../../py/emitinlinethumb.c ../../py/asmarm.c ../../py/emitnarm.c ../../py/asmxtensa.c ../../py/emitnxtensa.c ../../py/emitinlinextensa.c ../../py/formatfloat.c ../../py/parsenumbase.c ../../py/parsenum.c ../../py/emitglue.c ../../py/persistentcode.c ../../py/runtime.c ../../py/runtime_utils.c ../../py/scheduler.c ../../py/nativeglue.c ../../py/stackctrl.c ../../py/argcheck.c ../../py/warning.c ../../py/map.c ../../py/obj.c ../../py/objarray.c ../../py/objattrtuple.c ../../py/objbool.c ../../py/objboundmeth.c ../../py/objcell.c ../../py/objclosure.c ../../py/objcomplex.c ../../py/objdeque.c ../../py/objdict.c ../../py/objenumerate.c ../../py/objexcept.c ../../py/objfilter.c ../../py/objfloat.c ../../py/objfun.c ../../py/objgenerator.c ../../py/objgetitemiter.c ../../py/objint.c ../../py/objint_longlong.c ../../py/objint_mpz.c ../../py/objlist.c ../../py/objmap.c ../../py/objmodule.c ../../py/objobject.c ../../py/objpolyiter.c ../../py/objproperty.c ../../py/objnone.c ../../py/objnamedtuple.c ../../py/objrange.c ../../py/objreversed.c ../../py/objset.c ../../py/objsingleton.c ../../py/objslice.c ../../py/objstr.c ../../py/objstrunicode.c ../../py/objstringio.c ../../py/objtuple.c ../../py/objtype.c ../../py/objzip.c ../../py/opmethods.c ../../py/sequence.c ../../py/stream.c ../../py/binary.c ../../py/builtinimport.c ../../py/builtinevex.c ../../py/builtinhelp.c ../../py/modarray.c ../../py/modbuiltins.c ../../py/modcollections.c ../../py/modgc.c ../../py/modio.c ../../py/modmath.c ../../py/modcmath.c ../../py/modmicropython.c ../../py/modstruct.c ../../py/modsys.c ../../py/moduerrno.c ../../py/modthread.c ../../py/vm.c ../../py/bc.c ../../py/showbc.c ../../py/repl.c ../../py/smallint.c ../../py/frozenmod.c ../../extmod/moductypes.c ../../extmod/modujson.c ../../extmod/modure.c ../../extmod/moduzlib.c ../../extmod/moduheapq.c ../../extmod/modutimeq.c ../../extmod/moduhashlib.c ../../extmod/moducryptolib.c ../../extmod/modubinascii.c ../../extmod/virtpin.c ../../extmod/machine_mem.c ../../extmod/machine_pinbase.c ../../extmod/machine_signal.c ../../extmod/machine_pulse.c ../../extmod/machine_i2c.c ../../extmod/machine_spi.c ../../extmod/modussl_axtls.c ../../extmod/modussl_mbedtls.c ../../extmod/modurandom.c ../../extmod/moduselect.c ../../extmod/modwebsocket.c ../../extmod/modwebrepl.c ../../extmod/modframebuf.c ../../extmod/vfs.c ../../extmod/vfs_reader.c ../../extmod/vfs_posix.c ../../extmod/vfs_posix_file.c ../../extmod/vfs_fat.c ../../extmod/vfs_fat_diskio.c ../../extmod/vfs_fat_file.c ../../extmod/utime_mphal.c ../../extmod/uos_dupterm.c ../../lib/embed/abort_.c ../../lib/utils/printf.c main.c uart.c isr.c modmachine.c modlitex.c litex_leds.c litex_switches.c ../../lib/utils/stdout_helpers.c ../../lib/utils/interrupt_char.c ../../lib/utils/pyexec.c ../../lib/libc/string0.c ../../lib/mp-readline/readline.c ../../py/mpconfig.h mpconfigport.h >/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython/genhdr/qstr.i.last;
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
lm32-elf-newlib-gcc: error trying to exec 'cc1': execvp: No such file or directory
../../py/mkrules.mk:74: recipe for target '/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython/genhdr/qstr.i.last' failed
make: *** [/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython/genhdr/qstr.i.last] Error 1
make: *** Deleting file '/src/fpga/litex-buildenv/build/mimasv2_base_lm32/software/micropython/genhdr/qstr.i.last'
make: Leaving directory '/src/fpga/litex-buildenv/third_party/micropython/ports/fupy'
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ which lm32-elf-newlib-gcc
/src/fpga/litex-buildenv/build/conda/bin/lm32-elf-newlib-gcc
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda package -w $(which lm32-elf-newlib-gcc)
WARNING: The conda.install module is deprecated and will be removed in a future release.
/src/fpga/litex-buildenv/build/conda/bin/lm32-elf-newlib-gcc  timvideos::gcc-lm32-elf-newlib-5.4.0-20180626_181010
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ find build/conda -name "cc1"
build/conda/libexec/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/cc1
build/conda/libexec/gcc/lm32-elf/8.2.0/cc1
build/conda/libexec/gcc/or1k-elf/8.2.0/cc1
build/conda/pkgs/gcc-lm32-elf-nostdc-8.2.0-20181215_231127/libexec/gcc/lm32-elf/8.2.0/cc1
build/conda/pkgs/gcc-or1k-elf-nostdc-8.2.0_3006_g592479378c3-20181223_002632/libexec/gcc/or1k-elf/8.2.0/cc1
build/conda/pkgs/gcc-lm32-elf-nostdc-5.4.0-20180626_180158/libexec/gcc/lm32-elf/5.4.0/cc1
build/conda/pkgs/gcc_impl_linux-64-7.3.0-habb00fd_1/libexec/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/cc1
build/conda/pkgs/gcc-lm32-elf-nostdc-8.2.0-20190223_232653/libexec/gcc/lm32-elf/8.2.0/cc1
build/conda/pkgs/gcc-or1k-elf-nostdc-8.2.0_3006_g592479378c3-20181126_033222/libexec/gcc/or1k-elf/8.2.0/cc1
build/conda/pkgs/gcc-lm32-elf-nostdc-8.2.0-20181223_002106/libexec/gcc/lm32-elf/8.2.0/cc1
build/conda/pkgs/gcc-or1k-elf-nostdc-5.4.0_4334_g9310fdc97ee-20180626_180828/libexec/gcc/or1k-elf/5.4.0/cc1
build/conda/pkgs/gcc-or1k-elf-nostdc-8.2.0_3006_g592479378c3-20190223_233103/libexec/gcc/or1k-elf/8.2.0/cc1
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda package -w build/conda/libexec/gcc/lm32-elf/8.2.0/cc1
WARNING: The conda.install module is deprecated and will be removed in a future release.
build/conda/libexec/gcc/lm32-elf/8.2.0/cc1          timvideos::gcc-lm32-elf-nostdc-8.2.0-20190223_232653
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda package -w build/conda/libexec/gcc/or1k-elf/8.2.0/cc1
WARNING: The conda.install module is deprecated and will be removed in a future release.
build/conda/libexec/gcc/or1k-elf/8.2.0/cc1          timvideos::gcc-or1k-elf-nostdc-8.2.0_3006_g592479378c3-20190223_233103
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda upgrade gcc-${CPU}-elf-newlib
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /src/fpga/litex-buildenv/build/conda

  added / updated specs:
    - gcc-lm32-elf-newlib

The following packages will be UPDATED:

  gcc-lm32-elf-newl~                  5.4.0-20180626_181010 --> 8.2.0-20181223_002105

The following packages will be DOWNGRADED:

  binutils-lm32-elf                  2.31.1-20181223_002104 --> 2.31.1-20181215_231016
  binutils-or1k-elf                  2.31.1-20190223_232653 --> 2.31.1-20181223_002044
  binutils_linux-64                       2.31.1-h6176602_6 --> 2.31.1-h6176602_3
  flterm                  2.4_0029_g47d3b15-20190224_000016 --> 2.4_0024_gee93960-20181223_010808
  gcc-lm32-elf-nost~                  8.2.0-20190223_232653 --> 8.2.0-20181215_231127
  gcc-or1k-elf-nost~ 8.2.0_3006_g592479378c3-20190223_2331~ --> 8.2.0_3006_g592479378c3-20181223_002632
  gcc_linux-64                             7.3.0-h553295d_6 --> 7.3.0-h553295d_3
  gxx_linux-64                             7.3.0-h553295d_6 --> 7.3.0-h553295d_3

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ find build/conda -name "cc1"build/conda/libexec/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/cc1
build/conda/libexec/gcc/lm32-elf/8.2.0/cc1
build/conda/libexec/gcc/or1k-elf/8.2.0/cc1
build/conda/pkgs/gcc-lm32-elf-nostdc-8.2.0-20181215_231127/libexec/gcc/lm32-elf/8.2.0/cc1
build/conda/pkgs/gcc-or1k-elf-nostdc-8.2.0_3006_g592479378c3-20181223_002632/libexec/gcc/or1k-elf/8.2.0/cc1
build/conda/pkgs/gcc-lm32-elf-nostdc-5.4.0-20180626_180158/libexec/gcc/lm32-elf/5.4.0/cc1
build/conda/pkgs/gcc_impl_linux-64-7.3.0-habb00fd_1/libexec/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/cc1
build/conda/pkgs/gcc-lm32-elf-nostdc-8.2.0-20190223_232653/libexec/gcc/lm32-elf/8.2.0/cc1
build/conda/pkgs/gcc-or1k-elf-nostdc-8.2.0_3006_g592479378c3-20181126_033222/libexec/gcc/or1k-elf/8.2.0/cc1
build/conda/pkgs/gcc-lm32-elf-nostdc-8.2.0-20181223_002106/libexec/gcc/lm32-elf/8.2.0/cc1
build/conda/pkgs/gcc-or1k-elf-nostdc-5.4.0_4334_g9310fdc97ee-20180626_180828/libexec/gcc/or1k-elf/5.4.0/cc1
build/conda/pkgs/gcc-or1k-elf-nostdc-8.2.0_3006_g592479378c3-20190223_233103/libexec/gcc/or1k-elf/8.2.0/cc1
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ scripts/build-micropython.sh 
[...]
python -m litex.soc.tools.mkmscimg -f build/mimasv2_base_lm32//software/micropython/firmware.bin -o build/mimasv2_base_lm32//software/micropython/firmware.fbi
python mkimage.py \
    -Ob toolchain_path /opt/Xilinx/   \
    --override-gateware=build/mimasv2_base_lm32//gateware/top.bin \
    --override-bios=build/mimasv2_base_lm32//software/bios/bios.bin \
    --override-firmware=build/mimasv2_base_lm32//software/micropython/firmware.fbi \
    --output-file=build/mimasv2_base_lm32//image-gateware+bios+micropython.bin

Gateware @ 0x00000000 (    341436 bytes) build/mimasv2_base_lm32//gateware/top.bin                    - Xilinx FPGA Bitstream
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff aa 99 55 66 30 a1 00 07 20 00 31 a1 03 80 31 41 3d 00 31 61 09 ee 31 c2 04 00 10 93 30 e1 00 cf 30 c1 00 81 20 00 20 00 20 00 20 00 20 00 20 00
    BIOS @ 0x00080000 (     19692 bytes) build/mimasv2_base_lm32//software/bios/bios.bin              - LiteX BIOS with CRC
98 00 00 00 d0 00 00 00 78 01 20 08 38 21 00 00 d0 e1 00 00 e0 00 00 3b 34 00 00 00 34 00 00 00 e0 00 00 00 34 00 00 00 34 00 00 00 34 00 00 00 34 00 00 00 34 00 00 00 34 00 00 00 34 00 00 00
Firmware @ 0x00088000 (    246292 bytes) build/mimasv2_base_lm32//software/micropython/firmware.fbi   - HDMI2USB Firmware in FBI format (loaded into DRAM)
00 03 c2 0c ff d5 e6 e3 98 00 00 00 d0 00 00 00 78 01 40 00 38 21 00 00 d0 e1 00 00 e0 00 00 3b 34 00 00 00 34 00 00 00 e0 00 00 00 34 00 00 00 34 00 00 00 34 00 00 00 34 00 00 00 34 00 00 00
----------------------------------------
       Remaining space    1293804 bytes (9 Megabits, 1.23 Megabytes)
           Total space    2097152 bytes (16 Megabits, 2.00 Megabytes)

Flash image: build/mimasv2_base_lm32//image-gateware+bios+micropython.bin
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff aa 99 55 66 30 a1 00 07 20 00 31 a1 03 80 31 41 3d 00 31 61 09 ee 31 c2 04 00 10 93 30 e1 00 cf 30 c1 00 81 20 00 20 00 20 00 20 00 20 00 20 00
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda upgrade flterm
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /src/fpga/litex-buildenv/build/conda

  added / updated specs:
    - flterm

The following packages will be UPDATED:

  binutils-lm32-elf                  2.31.1-20181215_231016 --> 2.31.1-20181223_002104
  binutils-or1k-elf                  2.31.1-20181223_002044 --> 2.31.1-20190223_232653
  binutils_linux-64                       2.31.1-h6176602_3 --> 2.31.1-h6176602_6
  flterm                  2.4_0024_gee93960-20181223_010808 --> 2.4_0029_g47d3b15-20190224_000016
  gcc-lm32-elf-nost~                  8.2.0-20181215_231127 --> 8.2.0-20190223_232653
  gcc-or1k-elf-nost~ 8.2.0_3006_g592479378c3-20181223_0026~ --> 8.2.0_3006_g592479378c3-20190223_233103
  gcc_linux-64                             7.3.0-h553295d_3 --> 7.3.0-h553295d_6
  gxx_linux-64                             7.3.0-h553295d_3 --> 7.3.0-h553295d_6

The following packages will be DOWNGRADED:

  gcc-lm32-elf-newl~                  8.2.0-20181223_002105 --> 5.4.0-20180626_181010

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ make firmware-connect
flterm --port=/dev/ttyACM0 --speed=19200
[FLTERM] v2.4-29-g47d3b15 Starting...

>>> 
>>> 
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
 SoC BIOS / CPU: LM32 /  83MHz
(c) Copyright 2012-2018 Enjoy-Digital
(c) Copyright 2007-2018 M-Labs Limited
Built Feb 25 2019 17:16:11

BIOS CRC passed (4a527290)
Initializing SDRAM...
SDRAM now under hardware control
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Booting from flash...
Loading 246284 bytes from flash...
Executing booted program at 0x40000000
MicroPython v1.9.4-534-gd2bd404 on 2019-02-25; litex with lm32
>>> 
[FLTERM] Exiting...
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv/build/conda/pkgs$ ls -d gcc-lm32-elf-newlib-*
gcc-lm32-elf-newlib-5.4.0-20180626_181010          gcc-lm32-elf-newlib-8.2.0-20181223_002105
gcc-lm32-elf-newlib-5.4.0-20180626_181010.tar.bz2  gcc-lm32-elf-newlib-8.2.0-20181223_002105.tar.bz2
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv/build/conda/pkgs$ find gcc-lm32-elf-newlib-* -name "*cc1*"
gcc-lm32-elf-newlib-5.4.0-20180626_181010/libexec/gcc/lm32-elf/5.4.0/cc1plus
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv/build/conda/pkgs$
ewenmcneill commented 5 years ago

This seems to be the latest conda flterm upload, from just over a day ago:

https://anaconda.org/TimVideos/flterm

which leads to this list of packages:

https://anaconda.org/TimVideos/repo

The flterm, gcc, binutils, etc pretty much all seem to be uploaded 2019-02-23, so are pretty recent.

But the two (actually three) affected packages were last uploaded in 2018-12-23 or 2018-12-16:

The only other old packages are (a) vtr, (b) openocd, and (c) a couple of very old compilers:

I don't think the really old compilers have been used for ages (and maybe should be removed?), but I suspect vtr and openocd are fine, and not needing updates.

The rest of the packages in that TimVideos anaconda repo seem to have been uploaded in the last couple of days.

I'm currently still not clear where these -newlib packages are built for upload to conda, and thus why the -newlib ones are left out. I think most of the conda build packages are in https://github.com/timvideos/conda-hdmi2usb-packages, and maybe built by travis there? Eg,

https://github.com/timvideos/conda-hdmi2usb-packages/tree/master/gcc/newlib

for the -newlib variant. But neither that nor the "-nostdc" one:

https://github.com/timvideos/conda-hdmi2usb-packages/tree/master/gcc/nostdc

have been updated in several months, so something else must have forced the rebuild of everything else.

Ewen

Flterm :: Anaconda Cloud
Package repository for timvideos :: Anaconda Cloud
Gcc Or1K Elf Newlib :: Anaconda Cloud
Gcc Lm32 Elf Newlib :: Anaconda Cloud
Gcc Riscv32 Elf Newlib :: Anaconda Cloud
Gcc Lm32 Elf :: Anaconda Cloud
Gcc Or1K Elf :: Anaconda Cloud
GitHub
timvideos/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - timvideos/conda-hdmi2usb-packages
GitHub
timvideos/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - timvideos/conda-hdmi2usb-packages
GitHub
timvideos/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - timvideos/conda-hdmi2usb-packages
ewenmcneill commented 5 years ago

It looks like lm32 gcc/newlib and or1k gcc/newlib are supposed to be built by travis in https://github.com/timvideos/conda-hdmi2usb-packages/, like the rest of the packages that did get updated.

So presumably those builds are failing for some reason, and thus haven't been uploaded?

(vtr hasn't been rebuilt recently, because it's commented out; but I'm not sure what's happened with openocd -- maybe that's either not changed (and thus doesn't need rebuilding) or is also failing to build.)

Ewen

GitHub
timvideos/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - timvideos/conda-hdmi2usb-packages
ewenmcneill commented 5 years ago

It looks like the latest flterm does conflict with gcc-lm32-elf-newlib on direct dependencies, in particular:

so the "compatible" sets of dependencies are at this point pretty distinct. The reasons that 5.4.0 is installable is that its dependencies are super loose, rather than tied to specific versions.

And even if there wasn't that one, gcc-lm32-elf-newlib depends on gcc-lm32-elf-nostdc of a slightly older version, and that too has its own dependencies issues.

There's a chance that now that almost everything else is updated, Travis might be able to build gcc-lm32-elf-newlib and gcc-or1k-elf-newlib (since those wouldn't build until "conda install" of the -nostdc versions was available). But I'm not clear what would trigger Travis to try again (since, eg, usually the GitHub integration only triggers Travis on a commit).

So either way it looks like this might require some manual investigation/intervention by someone who can see the Travis logs from https://github.com/timvideos/conda-hdmi2usb-packages...

Ewen

PS: Possibly some of these versions should be specified as ">=" rather than precise versions? That'd potentially allow more upgrades without quite so much lock-step of versions. (But unfortunately that won't work for version numbers that depend on git repo commits... :-( )

(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv/build/c
onda/pkgs/flterm-2.4_0029_g47d3b15-20190224_000016$ jq ".depends" info/repodata_
record.json 
[
  "binutils_impl_linux-64 2.31.1 h6176602_1",
  "binutils_linux-64 2.31.1 h6176602_6",
  "gcc_impl_linux-64 7.3.0 habb00fd_1",
  "gcc_linux-64 7.3.0 h553295d_6",
  "gxx_impl_linux-64 7.3.0 hdf63c60_1",
  "gxx_linux-64 7.3.0 h553295d_6",
  "libgcc-ng 8.2.0 hdf63c60_1",
  "libstdcxx-ng 8.2.0 hdf63c60_1"
]
(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv/build/c
onda/pkgs/flterm-2.4_0029_g47d3b15-20190224_000016$ 
(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv
/build/conda/pkgs/gcc-lm32-elf-newlib-8.2.0-20181223_002105$ jq ".depends" info/repodata_record.json 
[
  "binutils-lm32-elf 2.31.1 20181215_231016",
  "binutils_impl_linux-64 2.31.1 h6176602_1",
  "binutils_linux-64 2.31.1 h6176602_3",
  "cloog 0.18.1 0",
  "gcc-lm32-elf-nostdc 8.2.0 20181215_231127",
  "gcc_impl_linux-64 7.3.0 habb00fd_1",
  "gcc_linux-64 7.3.0 h553295d_3",
  "gmp 6.1.2 h6c8ec71_1",
  "gxx_impl_linux-64 7.3.0 hdf63c60_1",
  "gxx_linux-64 7.3.0 h553295d_3",
  "isl 0.15 0",
  "libgcc-ng 8.2.0 hdf63c60_1",
  "libstdcxx-ng 8.2.0 hdf63c60_1",
  "mpc 1.1.0 h10f8cd9_1",
  "mpfr 4.0.1 hdf1c602_3"
]
(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv
/build/conda/pkgs/gcc-lm32-elf-newlib-8.2.0-20181223_002105$ 
(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv
/build/conda/pkgs/gcc-lm32-elf-nostdc-8.2.0-20181215_231127$ jq ".depends" info/
repodata_record.json 
[
  "binutils-lm32-elf 2.31.1 20181215_231016",
  "binutils_impl_linux-64 2.31.1 h6176602_1",
  "binutils_linux-64 2.31.1 h6176602_3",
  "cloog 0.18.1 0",
  "gcc_impl_linux-64 7.3.0 habb00fd_1",
  "gcc_linux-64 7.3.0 h553295d_3",
  "gmp 6.1.2 h6c8ec71_1",
  "gxx_impl_linux-64 7.3.0 hdf63c60_1",
  "gxx_linux-64 7.3.0 h553295d_3",
  "isl 0.15 0",
  "libgcc-ng 8.2.0 hdf63c60_1",
  "libstdcxx-ng 8.2.0 hdf63c60_1",
  "mpc 1.1.0 h10f8cd9_1",
  "mpfr 4.0.1 hdf1c602_3"
]
(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv
/build/conda/pkgs/gcc-lm32-elf-nostdc-8.2.0-20181215_231127$ 
(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv
/build/conda/pkgs/gcc-lm32-elf-newlib-5.4.0-20180626_181010$ jq ".depends" info/
repodata_record.json 
[
  "binutils-lm32-elf",
  "cloog",
  "gcc-lm32-elf-nostdc",
  "gmp >=4.3.2",
  "isl",
  "mpc >=0.8.1",
  "mpfr >=2.4.2"
]
(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv
/build/conda/pkgs/gcc-lm32-elf-newlib-5.4.0-20180626_181010$ 
GitHub
timvideos/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - timvideos/conda-hdmi2usb-packages
mithro commented 5 years ago

@ewenmcneill Thanks for the detailed report. The packages are being built at https://travis-ci.org/timvideos/conda-hdmi2usb-packages

The issue is that conda has been making changes faster than I can keep up with them, so the travis build has spent a lot of time red recently.

Travis CI - Test and Deploy Your Code with Confidence
mithro commented 5 years ago

It looks like the newlib gcc builds have been failing for a while now. PACKAGE=gcc/newlib TOOLCHAIN_ARCH=lm32

ewenmcneill commented 5 years ago

@mithro Yes, my guess is that the -newlib gcc builds have been failing for a couple of months, based on what I remember seeing in January (similar downgrades, but I didn't debug it, as I didn't have a reason to use the newer versions at that point), and the dates on the last successful uploads.

Thanks for the links to the https://travis-ci.org/timvideos/conda-hdmi2usb-packages build logs. I was trying to find Travis output, but not finding much via searching and didn't think to just try guessing the URL :-)

https://travis-ci.org/timvideos/conda-hdmi2usb-packages/jobs/498018878

(latest gcc-lm32-elf-newlib build), which is truncated (see raw log), suggests the current failure condition is "too much output":

The job exceeded the maximum log length, and has been terminated.

(literal text at the end of the "raw log")

So possibly some steps need to be built with output redirected to temporary files to avoid Travis CI logging them/counting them?

Particularly since gcc builds produce a lot of output (due to three build cycles), and adding newlib on top of that might just be too much.

Ewen

PS: Are there any notes on setting up Travis via GitHub to build a fork of (perhaps just selective packages), so I could experiment further with why that specific build is failing / if there's an easy fix? (Obviously it wouldn't be able to upload, but at least if I can get the build part running to completion we'd have an idea what to do to fix it.)

Travis CI - Test and Deploy Your Code with Confidence
Travis CI - Test and Deploy Your Code with Confidence
mithro commented 5 years ago

Should just be able to enable travis on your fork. I'm trying to make it work here -> https://travis-ci.org/mithro/conda-hdmi2usb-packages

Travis CI - Test and Deploy Your Code with Confidence
ewenmcneill commented 5 years ago

Looks like maximum log size is about 4MB, but they also need to produce output at least every 10 minutes. A suggested work around is travis_wait N, where N is the number of minutes that travis_wait should produce "comfort output" to keep the build running. They do have an example of travis_wait 30 (30 minutes), so presumably that's at least possible. Which means maybe the combination of I/O redirection and travis_wait 30 might be sufficient.

Based on the earlier job (https://travis-ci.org/timvideos/conda-hdmi2usb-packages/jobs/498018878) it looks like they hit the 4MB log file size around 12 minutes in, and the gcc-lm32-elf-nolibc ones are taking 20-30 minutes to build.

FWIW, I suspect rather than pinning gcc-${CPU}-elf-newlib to an exact matching gcc-${CPU}-elf-nolibc version, it might be best just to check the major/minor of the compiler version. So, eg, match gcc 8.20 if you can. That'll at least give the same path names to shared resources (eg, cc1), and would allow for less coupling between the builds.

If you don't find a solution easily soon, I might seen if I can conda build the -newlib ones on a separate test VM, and get some idea of the volume of output produced (and at which steps), the time taken, and whether it'll run to completion by itself at present (or also has other issues beyond log volume / build time). And then maybe try to reproduce that in travis too.

Ewen

Travis CI - Test and Deploy Your Code with Confidence
ewenmcneill commented 5 years ago

I've created https://github.com/timvideos/conda-hdmi2usb-packages/pull/63 to document how to do conda builds from https://github.com/timvideos/conda-hdmi2usb-packages locally in a VM (and where to find the Travis CI build logs for ease of reference).

I was able to build/install all three packages (bintuils, gcc/nostdc, gcc/newlib) in that order (build --check, build, install for each package before moving on to the next) with no issues (other than running out of disk space part way through and needing to expand the VM disk!). So the actual conda-build recipes seem to me to be fine. It looks like we're just running up against build ordering / version conflict issues, and Travis CI resource limits of various kinds which we need to manage.

From digging into how they're being built, my main observation is that possibly we want to rearrange how we build the toolchains for specific architectures, so that we build everything for one architecture in the same job.

Eg, instead of building lm32 in four separate Travis jobs, where the ordering that they happen to run/complete really matters for gcc/newlib as it's basically dependent on the gcc/nostdc one having (a) finished successfully, and (b) been installed into the build environment already, maybe we arrange it something like this:

TOOLCHAIN_ARCH=lm32 PACKAGES=binutils,gcc/nostdc,gcc/newlib,gdb

and then have that Travis CI job iterate (in script.sh or a wrapper around it), over all the packages that need to be built, building then installing them. That way each runs within a job that has already built/installed the relevant parts. And they inherently run with the same date/time stamp as well, so we get version lockstep.

The biggest issues with doing that are:

But it might be possible to solve both of those Travis CI environment issues, with some work arounds, which would then give us a more robust version lockstep build process.

(We'd also need to be able to build single packages, with just PACKAGE set, for the native programs; so we probably need to special case TOOLCHAIN_ARCH and PACKAGES being set to do the "loop over all packages" behaviour.)

Ewen

PS: For reference build times testing locally for me (copied from that pull request):

binutils build resources:

Total time: 0:02:00.4
CPU usage: sys=0:00:01.0, user=0:00:09.6
Maximum memory usage observed: 203.8M
Total disk usage observed (not including envs): 324.8K

gcc/nostdcc build resources:

Total time: 0:12:32.1
CPU usage: sys=0:00:09.1, user=0:03:38.6
Maximum memory usage observed: 535.2M
Total disk usage observed (not including envs): 731.7K

gcc/newlib build resources:

Total time: 0:27:00.9
CPU usage: sys=0:00:10.9, user=0:03:36.8
Maximum memory usage observed: 529.8M
Total disk usage observed (not including envs): 828.1K
GitHub
timvideos/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - timvideos/conda-hdmi2usb-packages
ewenmcneill commented 5 years ago

In this failed job it looks like the issue is conflicting gcc dependencies (possibly the native compiler?):

vagrant@conda:~/conda-hdmi2usb-packages$ ./conda-env.sh render gcc/newlib | grep 7.3.0 | sort
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - gcc_linux-64 7.3.0 h553295d_6
        - gcc_linux-64 7.3.0 h553295d_6
        - gcc_linux-64 7.3.0 h553295d_6
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - gxx_linux-64 7.3.0 h553295d_6
        - gxx_linux-64 7.3.0 h553295d_6
        - gxx_linux-64 7.3.0 h553295d_6
vagrant@conda:~/conda-hdmi2usb-packages$ 

Same in this build failure.

For variety, this one seems to be due to binutils dependencies:

vagrant@conda:~/conda-hdmi2usb-packages$ ./conda-env.sh render gcc/newlib | grep 2.31.1 | sort
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - binutils_linux-64 2.31.1 h6176602_6
        - binutils_linux-64 2.31.1 h6176602_6
        - binutils_linux-64 2.31.1 h6176602_6
        - binutils-lm32-elf 2.31.1 20190226_024939
        - binutils-lm32-elf 2.31.1 20190226_024939
vagrant@conda:~/conda-hdmi2usb-packages$

I suspect we probably also want to run conda render ${PACKAGE} or similar in the $TRAVIS_BUILD_DIR/.travis/after_failure.sh script (maybe at the start) to help with debugging these build conflicts.

My hunch is that one of the dependencies is being literally listed, and one is coming from resolved_packages. But maybe one of them is coming in via compiler('c').

Possibly this can be avoided by ensuring the conda environment is more fully updated before building? Since Miniconda is definitely (based on my testing today) an older snapshot that needs updating.

conda-get.sh definitely does update some conda things, but not the host build tools. Maybe we should explicitly install/update the host build tools?

Also of note, the gcc/newlib package seems to list host and build dependencies as runtime dependencies, from resolved_packages. Which might not be helping. Maybe those need to be in a different location in the file? (Same in gcc/nostdc though.)

Ewen

PS: Looks like we might also want to proactively use travis_wait (as I mentioned in https://github.com/timvideos/litex-buildenv/issues/123#issuecomment-467193755) to work around termination due to no log output (which happens in some other builds. Maybe travis_wait 45 as one of the wrappers in .travis/script.yml?

ewenmcneill commented 5 years ago

@mithro Looks like everything except for openocd built successfully in the most recent build of https://github.com/mithro/conda-hdmi2usb-packages.

openocd hasn't built for quite a long time due to what looks like an unrelated issue.

Well done on figuring out enough work arounds to get gcc/newlib to build successfully!

I'm assuming this needs pushing to https://github.com/timvideos/conda-hdmi2usb-packages and another Travis CI cycle to actually upload to https://anaconda.org/TimVideos/repo. But it seems like it's working well enough now that it's worth doing that, which would hopefully fix the original issue in this ticket (conflicting gcc/nostdc and gcc/newlib versions).

I'll have a wee look at our conda recipe for openocd locally and see if I can find an obvious problem (jinja2 rendering of variables, but it's unclear exactly why from the Travis CI logs).

Ewen

GitHub
mithro/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - mithro/conda-hdmi2usb-packages
GitHub
timvideos/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - timvideos/conda-hdmi2usb-packages
Package repository for timvideos :: Anaconda Cloud
ewenmcneill commented 5 years ago

Looks like openocd now has some of the things we were patching in:

[...]
==> git log -n1 <==

commit f21c12abecb9df244f147740166378ede7ea398e
Author: Moritz Fischer <moritz.fischer@ettus.com>
Date:   Mon Jan 21 09:24:12 2019 -0800
[...]
==> git describe --tags --dirty <==

v0.10.0-701-gf21c12abe
[...]
Applying patch: '/home/vagrant/conda-hdmi2usb-packages/openocd/digilent-arty.patch'
Patch level ambiguous, selecting least deep
Trying to apply patch as-is
INFO:conda_build.source:Trying to apply patch as-is
patching file tcl/board/digilent_arty.cfg
Applying patch: '/home/vagrant/conda-hdmi2usb-packages/openocd/digilent-arty-s7.patch'
The next patch would create the file tcl/board/arty_s7.cfg,
which already exists!  Assume -R? [n] n
Apply anyway? [n] n
Skipping patch.
1 out of 1 hunk ignored
Warning: failed to download source.  If building, will try again after downloading recipe dependencies.
Error was: 
Command '['/usr/bin/patch', '-p0', '--ignore-whitespace', '-i', '/home/vagrant/conda-hdmi2usb-packages/openocd/digilent-arty-s7.patch']' returned non-zero exit status 1.
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for openocd
INFO:conda_build.metadata:Attempting to finalize metadata for openocd
Error: Failed to render jinja template in /home/vagrant/conda-hdmi2usb-packages/openocd/meta.yaml:
'GIT_DESCRIBE_TAG' is undefined
vagrant@conda:~/conda-hdmi2usb-packages$ 

I'll see if I can figure out how close our patched version is to what's upstream now.

Ewen

ewenmcneill commented 5 years ago

The openocd arty_s7.cfg file seems to be identical to the one we were patching in, except for a few comments:

vagrant@conda:/tmp$ wget -O github https://raw.githubusercontent.com/ntfreak/openocd/master/tcl/board/arty_s7.cfg
--2019-02-27 16:58:56--  https://raw.githubusercontent.com/ntfreak/openocd/master/tcl/board/arty_s7.cfg
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.164.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.164.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1066 (1.0K) [text/plain]
Saving to: ‘github’

github              100%[===================>]   1.04K  --.-KB/s    in 0s      

2019-02-27 16:58:56 (23.7 MB/s) - ‘github’ saved [1066/1066]

vagrant@conda:/tmp$ patch -p 2 </home/vagrant/conda-hdmi2usb-packages/openocd/digilent-arty-s7.patch
patching file arty_s7.cfg
vagrant@conda:/tmp$ diff arty_s7.cfg github 
15,18d14
< # This configuration can only be used to load bitstreams either to the FPGA
< # via JTAG, or writing to an SPI flash via a jtagspi proxy. This configuration
< # cannot be used for debugging a target FPGA.
< #
vagrant@conda:/tmp$ 

so I think we just drop that patch. (That's not the actual openocd upstream we're using, but it looks like an official mirror onto GitHub; looks like the arty_s7.cfg was added by in October 2018, which is roughly when our openocd conda builds last worked...)

Interestingly it looks like there isn't an Arty board upstream, so we do still need that patch. Possibly we should try to submit our Arty (A7) one upstream, maybe renaming it to arty_a7?

Ewen

ewenmcneill commented 5 years ago

Also numato_mimas_a7.cfg has been merged upstream too, and seems to be identical:

vagrant@conda:/tmp$ wget -O github https://raw.githubusercontent.com/ntfreak/openocd/master/tcl/board/numato_mimas_a7.cfg
--2019-02-27 17:05:46--  https://raw.githubusercontent.com/ntfreak/openocd/master/tcl/board/numato_mimas_a7.cfg
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.164.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.164.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1255 (1.2K) [text/plain]
Saving to: ‘github’

github              100%[===================>]   1.23K  --.-KB/s    in 0s      

2019-02-27 17:05:46 (7.90 MB/s) - ‘github’ saved [1255/1255]

vagrant@conda:/tmp$ patch -p 2 </home/vagrant/conda-hdmi2usb-packages/openocd/numato-mimas_a7.patch 
patching file numato_mimas_a7.cfg
vagrant@conda:/tmp$ diff numato_mimas_a7.cfg github 
vagrant@conda:/tmp$ 

so we can drop that patch too.

And it looks like the cpld/xilinx-xc7.cfg additional codes that we had have also been merged upstream around October 2018:

vagrant@conda:/tmp/openocd$ git log -1 -p 988a002a30afe4facd116c97cba88b3793841046 >../github
vagrant@conda:/tmp/openocd$ cd ..
vagrant@conda:/tmp$ diff /home/vagrant/conda-hdmi2usb-packages/openocd/spartan7-idcodes.patch github 
0a1,17
> commit 988a002a30afe4facd116c97cba88b3793841046
> Author: William D. Jones <wjones@wdj-consulting.com>
> Date:   Thu Feb 22 11:30:18 2018 -0500
> 
>     xilinx-xc7: Add additional IDCODEs.
>     
>     Add/detect missing IDCODEs for the Spartan 7 family and Artix 25T
>     and Artix 12T.
>     
>     Change-Id: Ib6c83c5592e90df1eb8e715e79b279da9a95f9c6
>     Signed-off-by: William D. Jones <wjones@wdj-consulting.com>
>     Reviewed-on: http://openocd.zylin.com/4428
>     Reviewed-by: Robert Jördens
>     Tested-by: jenkins
>     Reviewed-by: Rohit Singh <rohit91.2008@gmail.com>
>     Reviewed-by: Matthias Welwarsky <matthias@welwarsky.de>
> 
2c19
< index d5824f8..4c0502c 100644
---
> index d5824f8a..4c0502c5 100644
5c22
< @@ -9,7 +9,15 @@
---
> @@ -9,7 +9,15 @@ if { [info exists CHIPNAME] } {
vagrant@conda:/tmp$ 

so we can drop that patch too.

One other patch is now applying with an offset, but for now I think it's okay:

Applying patch: '/home/vagrant/conda-hdmi2usb-packages/openocd/fpga-program.patch'
patching file tcl/cpld/xilinx-xc6s.cfg
patching file tcl/cpld/xilinx-xc7.cfg
Hunk #1 succeeded at 63 (offset 8 lines).
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Attempting to finalize metadata for openocd
INFO:conda_build.metadata:Attempting to finalize metadata for openocd

(That offset is presumably due to the other tcl/cpld/xilinx-xc7.cfg patch being merged, since it used to apply first rather than second.)

Ewen

mithro commented 5 years ago

Build happening at https://travis-ci.org/timvideos/conda-hdmi2usb-packages/builds/499100927 -- will probably need testing after it finishes.

Travis CI - Test and Deploy Your Code with Confidence
ewenmcneill commented 5 years ago

@mithro Great! I'm crossing my fingers it works out :-)

I have two pull requests to create once that's merged/built/uploaded/tested:

(both currently based on what is likely to be the previous master, as I created both branches a while ago; but hopefully they'd merge without conflicts). I won't make the pull requests now, because I don't want to risk interrupting that build/confusing the results of your current build (https://github.com/timvideos/litex-buildenv/issues/123#issuecomment-467733395). But I'm happy to create pull requests tomorrow if it seems good.

I can now conda build openocd locally (which took... a lot of tweaking), on an Ubuntu 18.04 Vagrant VM, so hopefully it'll build in Travis too. I've not tested the resulting build (only just got it to compile reliably...), but in theory it's upstream openocd plus the patches we have that weren't merged upstream.

The readme-updates are just more explanation of where to find things, a typo fix in the previous README update, and picking up your trick of using the git commit as the reference timestamp to use in local build attempts.

Ewen

GitHub
ewenmcneill/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - ewenmcneill/conda-hdmi2usb-packages
GitHub
ewenmcneill/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - ewenmcneill/conda-hdmi2usb-packages
ewenmcneill commented 5 years ago

I've just spotted that you also have a commit for openocd removing the patches that are upstream. And interestingly it looks like it all built in Travis from your (mithro) repo, apparently without the other changes I needed to make to the tooling to get it to build. (Possibly Travis is building it in a newer build environment than Ubuntu 18.04 LTS? In particular it looked to me like the pkg-config version was the issue for me, after a bunch of debugging.)

Ewen

ewenmcneill commented 5 years ago

Re https://github.com/timvideos/litex-buildenv/issues/123#issuecomment-467756418:

apparently without the other changes I needed to make to the tooling to get it to build ...openocd

On re-testing now, it turns out that the only important things to get it to build locally is ensuring that libtool and pkg-config are both installed in the host OS, which for some reason wasn't happening automatically with what looked like the same apt-get setup as .travis.yml. So I've added details of those to the README.md update branch. With those two host packages installed, and the conda automake/autoconf/pkg-config/libtool packages removed, I can build openocd here, with just your patch. I guess I never tried that particular combination without other partial attempts confusing it.

That's good news, because it means your current build will hopefully build everything successfully.

Of note:

https://github.com/mithro/conda-hdmi2usb-packages/blob/master/openocd/digilent-arty-s7.patch https://github.com/mithro/conda-hdmi2usb-packages/blob/master/openocd/numato-mimas_a7.patch https://github.com/mithro/conda-hdmi2usb-packages/blob/master/openocd/spartan7-idcodes.patch

as they've been merged upstream.

@mithro: If you agree, I could perhaps add that tidy up to my readme-updates branch, so we only get one more update/Travis CI build.

Ewen

GitHub
mithro/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - mithro/conda-hdmi2usb-packages
GitHub
mithro/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - mithro/conda-hdmi2usb-packages
GitHub
mithro/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - mithro/conda-hdmi2usb-packages
ewenmcneill commented 5 years ago

@mithro Test install attempt now that most of the timvideos Travis CI build is complete (and thus most of the packages have been updated). It looks pretty good. One needs to be wary of running conda update --all as it seems like we've got some dependencies on newer packages in pkgs/free which is a "lower priority" channel than pkgs/main, and if you allow it to go ahead, it'll swap those to the "higher priority" channel, which removes enough dependencies that we end up installing 5.4.0 compilers again. But other than that, it looks like we can now install a recent flterm and recent compiler builds simultaneously.

(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda list | grep timvideos
binutils-lm32-elf         2.31.1          20190227_054711    timvideos
binutils-or1k-elf         2.31.1          20190227_054711    timvideos
binutils-riscv32-elf      2.31.1          20190227_054711    timvideos
flterm                    2.4_0029_g47d3b15 20190226_054057    timvideos
gcc-lm32-elf-newlib       8.2.0           20190227_054711    timvideos
gcc-lm32-elf-nostdc       8.2.0           20190227_054711    timvideos
gcc-or1k-elf-newlib       8.2.0_3006_g592479378c3 20190227_054711    timvideos
gcc-or1k-elf-nostdc       8.2.0_3006_g592479378c3 20190227_054711    timvideos
icestorm                  0.0_0659_g8f61acd 20190227_054711    timvideos
nextpnr                   0.0.0_1879_g4c73061 20190227_054711    timvideos
openocd                   0.10.0_0554_g05e0d633b 20181016_152020    timvideos
yosys                     0.8_0326_g807b3c76 20190226_055019    timvideos
(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda upgrade --all --dry-run
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /src/fpga/litex-buildenv/build/conda

The following packages will be SUPERSEDED by a higher-priority channel:

  cloog                           pkgs/free::cloog-0.18.1-0 --> pkgs/main::cloog-0.18.0-0
  isl                                 pkgs/free::isl-0.15-0 --> pkgs/main::isl-0.12.2-0

The following packages will be DOWNGRADED:

  gcc-lm32-elf-newl~                  8.2.0-20190227_054711 --> 5.4.0-20180626_181010
  gcc-lm32-elf-nost~                  8.2.0-20190227_054711 --> 5.4.0-20180626_180158
  gcc-or1k-elf-newl~ 8.2.0_3006_g592479378c3-20190227_0547~ --> 5.4.0_4334_g9310fdc97ee-20180626_181158
  gcc-or1k-elf-nost~ 8.2.0_3006_g592479378c3-20190227_0547~ --> 5.4.0_4334_g9310fdc97ee-20180626_180828

Dry run. Exiting.

(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

scripts/dowload-env.sh for mimasv2 and arty visually look like they're happy too (eg, I didn't spot any obvious downgrading going on).

And openocd is the only thing which hasn't yet been built / uploaded to anaconda and isn't a build this month:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda list | grep timvideos | grep -v 201902
openocd                   0.10.0_0554_g05e0d633b 20181016_152020    timvideos
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

which will hopefully catch up pretty soon.

I'm going to try some test make clean / make gateware / scripts/build-micropython.sh now, just to double check that's working.

Ewen

ewenmcneill commented 5 years ago

Build completed, most packages show as updated at anacoda now (yosys and flterm currently still show yesterday's date, maybe cached index, and openocd is only partly showing as updated, so I still can't install the latest build via conda yet).

@mithro lm32 seems to work for MicroPython on Mimas V2, Arty (older openocd build uploading to it), and TinyFPGA BX (yesterday's yosys build compiling it), all using yesterday's flterm.

So other than wanting to re-test with the new openocd build on the Arty tomorrow, when it's visible for download via conda, and the README pull request/cleanup mentioned earlier, I think we're done!

Thanks for your help,

Ewen

PS: I haven't tried to test or1k today, but AFAIK it should be the same package build as before, just with better dependencies, and I can install the packages (see previous commit) without conflicts.


Successful test with Mimas v2 and these packages:

python $(which MimasV2Config.py) /dev/ttyACM0 build/mimasv2_base_lm32//image-gateware+bios+micropython.bin
****************************************
* Numato Lab Mimas V2 Configuration Tool *
****************************************
Micron M25P16 SPI Flash detected
Loading file build/mimasv2_base_lm32//image-gateware+bios+micropython.bin...
Erasing flash sectors...
Writing to flash 100% complete...
Verifying flash contents...
Flash verification successful...
Booting FPGA...
Done.
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda list | grep timvideos
binutils-lm32-elf         2.31.1          20190227_054711    timvideos
binutils-or1k-elf         2.31.1          20190227_054711    timvideos
binutils-riscv32-elf      2.31.1          20190227_054711    timvideos
flterm                    2.4_0029_g47d3b15 20190226_054057    timvideos
gcc-lm32-elf-newlib       8.2.0           20190227_054711    timvideos
gcc-lm32-elf-nostdc       8.2.0           20190227_054711    timvideos
gcc-or1k-elf-newlib       8.2.0_3006_g592479378c3 20190227_054711    timvideos
gcc-or1k-elf-nostdc       8.2.0_3006_g592479378c3 20190227_054711    timvideos
icestorm                  0.0_0659_g8f61acd 20190227_054711    timvideos
nextpnr                   0.0.0_1879_g4c73061 20190227_054711    timvideos
openocd                   0.10.0_0554_g05e0d633b 20181016_152020    timvideos
yosys                     0.8_0326_g807b3c76 20190226_055019    timvideos
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ make firmware-connect
flterm --port=/dev/ttyACM0 --speed=19200
[FLTERM] v2.4-29-g47d3b15 Starting...

>>> 
>>> 
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
 SoC BIOS / CPU: LM32 /  83MHz
(c) Copyright 2012-2018 Enjoy-Digital
(c) Copyright 2007-2018 M-Labs Limited
Built Feb 27 2019 21:33:41

BIOS CRC passed (6589be9e)
Initializing SDRAM...
SDRAM now under hardware control
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Booting from flash...
Loading 246284 bytes from flash...
Executing booted program at 0x40000000
MicroPython v1.9.4-534-gd2bd404 on 2019-02-27; litex with lm32
>>> 
[FLTERM] Exiting...
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

Successful test with Arty A7, although still only with older openocd, as currently https://anaconda.org/timvideos/openocd sitll shows "last update" of 4 months ago, even though https://anaconda.org/timvideos/openocd shows openocd updated 2019-02-27 (today). So I'm guessing there's some caching within anacoda still to catch up there...

(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ make gateware-load
openocd -f board/digilent_arty.cfg -c "init; pld load 0 build/arty_base_lm32//gateware/top.bit; exit"
Open On-Chip Debugger 0.10.0+dev-00554-g05e0d633b-dirty (2018-10-16-15:23)
Licensed under GNU GPL v2
For bug reports, read
    http://openocd.org/doc/doxygen/bugs.html
none separate
adapter speed: 10000 kHz
Info : auto-selecting first available session transport "jtag". To override use 'transport select <transport>'.
jtagspi_program
Info : ftdi: if you experience problems at higher adapter clocks, try the command "ftdi_tdo_sample_edge falling"
Info : clock speed 10000 kHz
Info : JTAG tap: xc7.tap tap/device found: 0x0362d093 (mfg: 0x049 (Xilinx), part: 0x362d, ver: 0x0)
loaded file build/arty_base_lm32//gateware/top.bit to pld device 0 in 1s 833693us
(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda list | grep timvideos
binutils-lm32-elf         2.31.1          20190227_054711    timvideos
binutils-or1k-elf         2.31.1          20190227_054711    timvideos
binutils-riscv32-elf      2.31.1          20190227_054711    timvideos
flterm                    2.4_0029_g47d3b15 20190226_054057    timvideos
gcc-lm32-elf-newlib       8.2.0           20190227_054711    timvideos
gcc-lm32-elf-nostdc       8.2.0           20190227_054711    timvideos
gcc-or1k-elf-newlib       8.2.0_3006_g592479378c3 20190227_054711    timvideos
gcc-or1k-elf-nostdc       8.2.0_3006_g592479378c3 20190227_054711    timvideos
icestorm                  0.0_0659_g8f61acd 20190227_054711    timvideos
nextpnr                   0.0.0_1879_g4c73061 20190227_054711    timvideos
openocd                   0.10.0_0554_g05e0d633b 20181016_152020    timvideos
yosys                     0.8_0326_g807b3c76 20190226_055019    timvideos
(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ make firmware-load
[...]
python -m litex.soc.tools.mkmscimg -f firmware.bin -o firmware.fbi
make[1]: Leaving directory '/src/fpga/litex-buildenv/build/arty_base_lm32/software/firmware'

real    0m2.556s
user    0m2.308s
sys 0m0.271s
flterm --port=/dev/ttyUSB1 --kernel=build/arty_base_lm32//software/micropython/firmware.bin --speed=115200
[FLTERM] v2.4-29-g47d3b15 Starting...

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
 SoC BIOS / CPU: LM32 / 100MHz
(c) Copyright 2012-2018 Enjoy-Digital
(c) Copyright 2007-2018 M-Labs Limited
Built Feb 27 2019 21:41:03

BIOS CRC passed (d43789dd)
Initializing SDRAM...
SDRAM now under software control
Read leveling:
m0, b0: |11111111111100000000000000000000| delays: 06+-06
m0, b1: |00000000000000011111111111100000| delays: 21+-06
m0, b2: |00000000000000000000000000000001| delays: 31+-00
m0, b3: |00000000000000000000000000000000| delays: -
m0, b4: |00000000000000000000000000000000| delays: -
m0, b5: |00000000000000000000000000000000| delays: -
m0, b6: |00000000000000000000000000000000| delays: -
m0, b7: |00000000000000000000000000000000| delays: -
best: m0, b0 delays: 06+-06
m1, b0: |11111111111100000000000000000000| delays: 06+-06
m1, b1: |00000000000000011111111111100000| delays: 21+-06
m1, b2: |00000000000000000000000000000011| delays: 31+-01
m1, b3: |00000000000000000000000000000000| delays: -
m1, b4: |00000000000000000000000000000000| delays: -
m1, b5: |00000000000000000000000000000000| delays: -
m1, b6: |00000000000000000000000000000000| delays: -
m1, b7: |00000000000000000000000000000000| delays: -
best: m1, b0 delays: 06+-06
SDRAM now under hardware control
Memtest OK
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
[FLTERM] Received firmware download request from the device.
[FLTERM] Uploading kernel (246284 bytes)...
[FLTERM] Upload complete (7.7KB/s).
[FLTERM] Booting the device.
[FLTERM] Done.
Executing booted program at 0x40000000
MicroPython v1.9.4-534-gd2bd404 on 2019-02-27; litex with lm32
>>> 
[FLTERM] Exiting...
(LX P=arty T=base F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

TinyFPGA BX:

(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda list | grep timvideos
binutils-lm32-elf         2.31.1          20190227_054711    timvideos
binutils-or1k-elf         2.31.1          20190227_054711    timvideos
binutils-riscv32-elf      2.31.1          20190227_054711    timvideos
flterm                    2.4_0029_g47d3b15 20190226_054057    timvideos
gcc-lm32-elf-newlib       8.2.0           20190227_054711    timvideos
gcc-lm32-elf-nostdc       8.2.0           20190227_054711    timvideos
gcc-or1k-elf-newlib       8.2.0_3006_g592479378c3 20190227_054711    timvideos
gcc-or1k-elf-nostdc       8.2.0_3006_g592479378c3 20190227_054711    timvideos
icestorm                  0.0_0659_g8f61acd 20190227_054711    timvideos
nextpnr                   0.0.0_1879_g4c73061 20190227_054711    timvideos
openocd                   0.10.0_0554_g05e0d633b 20181016_152020    timvideos
yosys                     0.8_0326_g807b3c76 20190226_055019    timvideos
(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ make image-flash
[...]
Flash image: build/tinyfpga_bx_base_lm32.minimal//image-gateware+bios+micropython.bin
ff 00 00 ff 7e aa 99 7e 51 00 01 05 92 00 21 62 03 67 72 01 10 82 00 00 11 00 01 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
tinyprog --program-image build/tinyfpga_bx_base_lm32.minimal//image-gateware+bios+micropython.bin

    TinyProg CLI
    ------------
    Using device id 1d50:6130
    Only one board with active bootloader, using it.

    Programming /dev/ttyACM0 with build/tinyfpga_bx_base_lm32.minimal//image-gateware+bios+micropython.bin
    Programming at addr 028000
    Waking up SPI flash
    442884 bytes to program
    Programming and Verifying: 100%|█████████| 443k/443k [00:10<00:00, 41.6kB/s]
    Success!

(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ make firmware-connect
flterm --port=/dev/ttyUSB0 --speed=115200
[FLTERM] v2.4-29-g47d3b15 Starting...
Executing booted program at 0x20058008
MicroPython v1.9.4-534-gd2bd404 on 2019-02-27; litex with lm32
>>> 
        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
 SoC BIOS / CPU: LM32 /  16MHz
(c) Copyright 2012-2018 Enjoy-Digital
(c) Copyright 2007-2018 M-Labs Limited
Built Feb 27 2019 22:00:23

BIOS CRC passed (b297661e)
Booting from serial...
Press Q or ESC to abort boot completely.
sL5DdSMmkekro
Timeout
Booting from flash...
Executing booted program at 0x20058008
MicroPython v1.9.4-534-gd2bd404 on 2019-02-27; litex with lm32
>>> 
>>> 
[FLTERM] Exiting...
(LX P=tinyfpga_bx.minimal F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 
Openocd :: Anaconda Cloud
mithro commented 5 years ago

Can you send your pull requests and I'll merge them?

ewenmcneill commented 5 years ago

@mithro https://github.com/timvideos/conda-hdmi2usb-packages/pull/64.

There's only one pull request, as (a) I added the "remove unused openocd patches" into that readme update pull request (to reduce Travis CI build runs), and (b) the rest of my openocd build hacking turned out to be unnecessary if pkg-config and libtool are explicitly installed in the host OS. (I've attached a diff of that openocd build hacking to the pull request for future reference, but the openocd changes aren't needed to build; openocd builds just fine here with just https://github.com/timvideos/conda-hdmi2usb-packages/pull/64. )

Of note, even though Travis CI apparently uploaded the new openocd build (see end), I still don't see it on anaconda / can't install it here. So maybe that upload got lost.

The yosys build, sdcc build, and flterm build all seem similarly affected -- 20190227 package version uploaded to anaconda, but not visible, for me at least, at https://anaconda.org/TimVideos/repo, even 12 hours later. But those non-openocd ones don't even show a new "last uploaded" date in the top index. (Only openocd is a really old build from 2018; the rest are 20190226 builds, so missing one doesn't matter much.)

Hopefully today's build, when you merge https://github.com/timvideos/conda-hdmi2usb-packages/pull/64, will manage to upload everything successfully and sort out the last stragglers on the anaconda channel.

Ewen

Package repository for timvideos :: Anaconda Cloud
mithro commented 5 years ago

user.exceeded_storage_quota - Going to clean up some stuff now.

ewenmcneill commented 5 years ago

@mithro Running out of space would do it! Curiously even though the build did finish okay, it still looks like some packages are only on the earlier uploads. Particularly our binutils:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda list | grep timvideos | grep -v 204219
binutils-lm32-elf         2.31.1          20190227_054711    timvideos
binutils-or1k-elf         2.31.1          20190227_054711    timvideos
binutils-riscv32-elf      2.31.1          20190227_054711    timvideos
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

all of which show last upload 18 hours ago (https://anaconda.org/timvideos/binutils-lm32-elf, https://anaconda.org/timvideos/binutils-or1k-elf, https://anaconda.org/timvideos/binutils-riscv32-elf).

But I've been able to (carefully, see below) update everything else to the current build:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda list | grep timvideos | grep -v binutils
flterm                    2.4_0029_g47d3b15 20190227_204219    timvideos
gcc-lm32-elf-newlib       8.2.0           20190227_204219    timvideos
gcc-lm32-elf-nostdc       8.2.0           20190227_204219    timvideos
gcc-or1k-elf-newlib       8.2.0_3006_g592479378c3 20190227_204219    timvideos
gcc-or1k-elf-nostdc       8.2.0_3006_g592479378c3 20190227_204219    timvideos
icestorm                  0.0_0659_g8f61acd 20190227_204219    timvideos
nextpnr                   0.0.0_1879_g4c73061 20190227_204219    timvideos
openocd                   0.10.0_0701_gf21c12abe 20190227_204219    timvideos
yosys                     0.8_0326_g807b3c76 20190227_204219    timvideos
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda list | grep timvideos | grep -v binutils | grep -v 20190227_204219
(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

so that's good.

The main remaining weirdness in dependencies is that it looks like the upstream conda binutils_linux-64 for the host seems to have been updated to 7.20 / 8.20, and due to our exact dependencies that means that some of our packages want to downgrade quite a bit:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda update --dry-run --all
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /src/fpga/litex-buildenv/build/conda

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    binutils_impl_linux-64-2.28.1|       had2808c_3        16.1 MB
    binutils_linux-64-7.2.0    |      had2808c_27           8 KB
    gcc_impl_linux-64-7.2.0    |       habb00fd_3        72.4 MB
    gcc_linux-64-7.2.0         |      h550dcbe_27           9 KB
    gxx_impl_linux-64-7.2.0    |       hdf63c60_3        18.6 MB
    gxx_linux-64-7.2.0         |      h550dcbe_27           8 KB
    ------------------------------------------------------------
                                           Total:       107.1 MB

The following packages will be UPDATED:

  binutils_linux-64                       2.31.1-h6176602_6 --> 7.2.0-had2808c_27

The following packages will be DOWNGRADED:

  binutils_impl_lin~                      2.31.1-h6176602_1 --> 2.28.1-had2808c_3
  flterm                  2.4_0029_g47d3b15-20190227_204219 --> 2.4_24_gee93960-0
  gcc_impl_linux-64                        7.3.0-habb00fd_1 --> 7.2.0-habb00fd_3
  gcc_linux-64                             7.3.0-h553295d_6 --> 7.2.0-h550dcbe_27
  gxx_impl_linux-64                        7.3.0-hdf63c60_1 --> 7.2.0-hdf63c60_3
  gxx_linux-64                             7.3.0-h553295d_6 --> 7.2.0-h550dcbe_27

Dry run. Exiting.

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

so currently conda upgrade --all is still unwise.

It looks like there's a bunch of binutils packages here:

https://repo.anaconda.com/pkgs/main/linux-64

which then depend on binutils_impl_linux-64 for specific releases, so maybe we have to, eg, update our binutils_linux-64 dependency to be a binutils_impl_linux-64 dependency to match what seems to have happened within Conda? (Or depend on binutils_linux-64 of the same version as the compiler.)

It looks like the remaining trigger for this might be our flterm dependencies, as that's the last thing that it wants to downgrade, and that does depend on binutils_linux-64 2.31.1 h6176602_6`:

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda info flterm 2>/dev/null | grep -A 30 "flterm 2.4_0029_g47d3b15 20190227_204219" | grep -A 9 dependencies:
dependencies:
    binutils_impl_linux-64 2.31.1 h6176602_1
    binutils_linux-64 2.31.1 h6176602_6
    gcc_impl_linux-64 7.3.0 habb00fd_1
    gcc_linux-64 7.3.0 h553295d_6
    gxx_impl_linux-64 7.3.0 hdf63c60_1
    gxx_linux-64 7.3.0 h553295d_6
    libgcc-ng 8.2.0 hdf63c60_1
    libstdcxx-ng 8.2.0 hdf63c60_1

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

This seems to be true even though flterm did get uploaded an hour ago.

Possibly due to https://github.com/timvideos/conda-hdmi2usb-packages/blob/master/flterm/meta.yaml having host and build dependencies as runtime dependencies, thus with exact version locks? Maybe we want to remove those strict dependencies from flterm, since surely it doesn't need much version specific?

Ewen

Binutils Lm32 Elf :: Anaconda Cloud
Binutils Or1K Elf :: Anaconda Cloud
Binutils Riscv32 Elf :: Anaconda Cloud
main/linux-64
GitHub
timvideos/conda-hdmi2usb-packages
Conda build recipes for the toolchains needed by LiteX / MiSoC firmware - timvideos/conda-hdmi2usb-packages
ewenmcneill commented 5 years ago

@mithro Re https://github.com/timvideos/conda-hdmi2usb-packages/pull/65, it might be worth considering if we still want to simplify the -newlib dependencies further for the same reason before merging that.

Because I don't see an obvious reason why we need to, eg, depend on the binutils that the host had for its compiler at its runtime in the runtime of our build:

vagrant@conda:~/conda-hdmi2usb-packages$ ./conda-env.sh render gcc/newlib | grep -A 99 requirements:  | grep -B 99 about: 
requirements:
    build:
        - gxx_linux-64 7.3.0 h553295d_6
        - binutils_linux-64 2.31.1 h6176602_6
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - gcc_linux-64 7.3.0 h553295d_6
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - libgcc-ng 8.2.0 hdf63c60_1
    host:
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - libgcc-ng 8.2.0 hdf63c60_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - binutils-lm32-elf 2.31.1 20190226_024939
        - binutils_linux-64 2.31.1 h6176602_6
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - gmp 6.1.2 h6c8ec71_1
        - gcc_linux-64 7.3.0 h553295d_6
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - isl 0.15 0
        - mpfr 4.0.1 hdf1c602_3
        - cloog 0.18.1 0
        - gxx_linux-64 7.3.0 h553295d_6
        - mpc 1.1.0 h10f8cd9_1
        - gcc-lm32-elf-nostdc 8.2.0 20190226_024939
    run:
        - binutils-lm32-elf 2.31.1 20190226_024939
        - binutils_linux-64 2.31.1 h6176602_6
        - gxx_linux-64 7.3.0 h553295d_6
        - mpfr 4.0.1 hdf1c602_3
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - gcc_linux-64 7.3.0 h553295d_6
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - gmp 6.1.2 h6c8ec71_1
        - gcc-lm32-elf-nostdc 8.2.0 20190226_024939
        - mpc 1.1.0 h10f8cd9_1
        - libgcc-ng 8.2.0 hdf63c60_1
        - cloog 0.18.1 0
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - isl 0.15 0
about:
vagrant@conda:~/conda-hdmi2usb-packages$ 

That list looks (a) rather long, and (b) rather more specific that seems ideal. (But at least after conda upgrade --all it's less conflicting...)

Ewen

ewenmcneill commented 5 years ago

FTR, the gcc/nostdc resulting runtime requirements are somewhat shorter:

vagrant@conda:~/conda-hdmi2usb-packages$ ./conda-env.sh render gcc/nostdc | grep -A 99 requirements:  | grep -B 99 about: 
requirements:
    build:
        - binutils_linux-64 2.31.1 h6176602_6
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - gxx_linux-64 7.3.0 h553295d_6
        - libgcc-ng 8.2.0 hdf63c60_1
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - gcc_linux-64 7.3.0 h553295d_6
    host:
        - libgcc-ng 8.2.0 hdf63c60_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - binutils-lm32-elf 2.31.1 20190226_024939
        - gmp 6.1.2 h6c8ec71_1
        - isl 0.15 0
        - mpfr 4.0.1 hdf1c602_3
        - cloog 0.18.1 0
        - mpc 1.1.0 h10f8cd9_1
    run:
        - mpfr 4.0.1 hdf1c602_3
        - gmp 6.1.2 h6c8ec71_1
        - libgcc-ng 8.2.0 hdf63c60_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - isl 0.15 0
        - cloog 0.18.1 0
        - binutils-lm32-elf 2.31.1 20190226_024939
        - mpc 1.1.0 h10f8cd9_1
about:
vagrant@conda:~/conda-hdmi2usb-packages$ 

and more sensible. So I'm not sure exactly what's causing the gcc/newlib ones to get so out of control, but the gcc/newlib ones are still too specific IMHO.

Ewen

ewenmcneill commented 5 years ago

@mithro FYI, removing the resolved_packages['host'] from gcc/newlib leads to a much shorter, more flexible, list of dependencies.

vagrant@conda:~/conda-hdmi2usb-packages$ ./conda-env.sh render gcc/newlib >/tmp/gcc-newlib.dep.before
vagrant@conda:~/conda-hdmi2usb-packages$ vi gcc/newlib/meta.yaml 
vagrant@conda:~/conda-hdmi2usb-packages$ git diff
diff --git a/gcc/newlib/meta.yaml b/gcc/newlib/meta.yaml
index aa7250b..6c83adb 100644
--- a/gcc/newlib/meta.yaml
+++ b/gcc/newlib/meta.yaml
@@ -53,9 +53,6 @@ requirements:
   run:
     - binutils-{{ environ.get('TOOLCHAIN_ARCH') }}-elf
     - gcc-{{ environ.get('TOOLCHAIN_ARCH') }}-elf-nostdc {{ version }}.*
-  {% for package in resolved_packages('host') %}
-    - {{ package }}
-  {% endfor %}

 about:
   home: https://gcc.gnu.org/
vagrant@conda:~/conda-hdmi2usb-packages$ ./conda-env.sh render gcc/newlib >/tmp/gcc-newlib.dep.after
vagrant@conda:~/conda-hdmi2usb-packages$ diff /tmp/gcc-newlib.dep.before /tmp/gcc-newlib.dep.after
35c35,37
<         - gxx_impl_linux-64 7.3.0 hdf63c60_1
---
>         - gcc_linux-64 7.3.0 h553295d_6
>         - gxx_linux-64 7.3.0 h553295d_6
>         - binutils_linux-64 2.31.1 h6176602_6
40,42c42
<         - gxx_linux-64 7.3.0 h553295d_6
<         - gcc_linux-64 7.3.0 h553295d_6
<         - binutils_linux-64 2.31.1 h6176602_6
---
>         - gxx_impl_linux-64 7.3.0 hdf63c60_1
60,74c60,65
<         - mpfr 4.0.1 hdf1c602_3
<         - libgcc-ng 8.2.0 hdf63c60_1
<         - binutils_impl_linux-64 2.31.1 h6176602_1
<         - gxx_linux-64 7.3.0 h553295d_6
<         - gcc_linux-64 7.3.0 h553295d_6
<         - mpc 1.1.0 h10f8cd9_1
<         - gmp 6.1.2 h6c8ec71_1
<         - gxx_impl_linux-64 7.3.0 hdf63c60_1
<         - binutils_linux-64 2.31.1 h6176602_6
<         - gcc-lm32-elf-nostdc 8.2.0 20190226_024939
<         - binutils-lm32-elf 2.31.1 20190226_024939
<         - isl 0.15 0
<         - libstdcxx-ng 8.2.0 hdf63c60_1
<         - gcc_impl_linux-64 7.3.0 habb00fd_1
<         - cloog 0.18.1 0
---
>         - binutils-lm32-elf
>         - libstdcxx-ng >=7.3.0
>         - mpfr >=4.0.1,<5.0a0
>         - gmp >=6.1.2
>         - gcc-lm32-elf-nostdc 8.2.0.*
>         - libgcc-ng >=7.3.0
vagrant@conda:~/conda-hdmi2usb-packages$ 
vagrant@conda:~/conda-hdmi2usb-packages$ grep -A 99 requirements: /tmp/gcc-newlib.dep.after | grep -B 99 about:
requirements:
    build:
        - gcc_linux-64 7.3.0 h553295d_6
        - gxx_linux-64 7.3.0 h553295d_6
        - binutils_linux-64 2.31.1 h6176602_6
        - libgcc-ng 8.2.0 hdf63c60_1
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
    host:
        - binutils_impl_linux-64 2.31.1 h6176602_1
        - libgcc-ng 8.2.0 hdf63c60_1
        - libstdcxx-ng 8.2.0 hdf63c60_1
        - binutils-lm32-elf 2.31.1 20190226_024939
        - binutils_linux-64 2.31.1 h6176602_6
        - gcc_impl_linux-64 7.3.0 habb00fd_1
        - gmp 6.1.2 h6c8ec71_1
        - gcc_linux-64 7.3.0 h553295d_6
        - gxx_impl_linux-64 7.3.0 hdf63c60_1
        - isl 0.15 0
        - mpfr 4.0.1 hdf1c602_3
        - cloog 0.18.1 0
        - gxx_linux-64 7.3.0 h553295d_6
        - mpc 1.1.0 h10f8cd9_1
        - gcc-lm32-elf-nostdc 8.2.0 20190226_024939
    run:
        - binutils-lm32-elf
        - libstdcxx-ng >=7.3.0
        - mpfr >=4.0.1,<5.0a0
        - gmp >=6.1.2
        - gcc-lm32-elf-nostdc 8.2.0.*
        - libgcc-ng >=7.3.0
about:
vagrant@conda:~/conda-hdmi2usb-packages$ 

And it seems to me that the gcc-lm32-elf-nostdc 8.2.0.* is going to transitively drag in the right things anyway.

Thoughts?

Ewen

ewenmcneill commented 5 years ago

FTR, after some fiddling with conda dependency checking, it appears the main reason conda upgrade -all --dry-run wanted to downgrade binutils is that the main conda gxx_linux-64 and gcc_linux-64 is broken for gcc 8.2.0 / gxx 8.2.0, because it's missing key dependencies.

Eg,

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ conda install --dry-run gxx_linux-64=8.2.0
Collecting package metadata: done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - gxx_linux-64=8.2.0 -> gcc_linux-64==8.2.0=h218040c_3 -> gcc_impl_linux-64=8.2.0
  - gxx_linux-64=8.2.0 -> gxx_impl_linux-64=8.2.0

Current channels:

  - https://conda.anaconda.org/timvideos/linux-64
  - https://conda.anaconda.org/timvideos/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/free/linux-64
  - https://repo.anaconda.com/pkgs/free/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

(LX P=mimasv2 F=micropython) ewen@parthenon:/src/fpga/litex-buildenv$ 

And the 8.2.0 g??_impl_linux-64 packages are indeed completely missing from https://repo.anaconda.com/pkgs/main/linux-64/.

However now that we've made our dependencies more relaxed, I was able to upgrade conda binutils_linux-64 manually (by specifying an 8.2.0 version), which removed gcc_linux-64 and gxx_linux-64, but then everything else was happy, including conda upgrade --dry-run --all.

I think we actually didn't need the conda native gcc / gxx (or at least scripts/download-env.sh wasn't actually downloading / installing them again), they were just being dragged in by having build-time dependencies stuck in as run time dependencies. So in theory that might have cleaned up my conda environment. (I'm not really sure if we need binutils_linux-64 either, but I've stopped fiddling for tonight, now that it doesn't want to downgrade things.)

FTR, https://github.com/conda/conda/issues/2361 had some useful hints, although conda now want conda info to be:

conda search '<package>=<version>=<build_num>' --info

(and it produces the same output, but without the warning about conda info being deprecated...).

for PACKAGE in $(conda list | awk '/timvideos/ { print $1 "=" $2 "=" $3; }'); do conda search "${PACKAGE}" --info; done

was also useful for checking our dependencies, which now seem pretty reasonable (which is what led me to realising there was a conda linux-64 dependency issue).

Ewen

main/linux-64
ewenmcneill commented 5 years ago

Closing ticket, since I think we've solved the original problem (inability to install current versions of various TimVideos conda packages simultaneously). It's possible one more build would get everything fully in sync (there's still a variety of build uploads as some CI uploads failed due to HTTPS/proxy issues). But that can happen at some later point without this ticket.

Ewen