Docker build method fails beyond AmigaTestKit-v1.7

[MatrixCat3D]

Trying to use :

docker run -e COMMIT=testkit-v1.8 -e MAKE_OPTS=-j4 -v $(pwd):/output --rm -ti rjnorthrow/atk

...(or later versions) results in:

...
CC video.o
VASM ptplayer/ptplayer.o
vasmm68k_mot -Felf ptplayer/ptplayer.asm -o ptplayer/ptplayer.o
make: vasmm68k_mot: Command not found
make: *** [../base/Rules.mk:65: ptplayer/ptplayer.o] Error 127
make: *** Waiting for unfinished jobs....

This is when running rjnorthrow/atk [554c6212d705] on Docker version 18.06.3-ce, build d7080c1

It looks like v1.3-1.7 do not use ptplayer which is why there is no need/dependecy on vasmm68k_mot - git history confirms the testkit/Makefile had ptplayer added 2020-06-03

[rjnorthrow]

Updated both rjnorthrow/m68k-toolchain [c5e97ed0] (to use GCC-9.3.0, binutils-2.35 and build vasm) and rjnorthrow/atk [00922ddf] (to build zopfli and inflate). The build now successfully creates the AmigaTestKit.exe and AmigaTestKit.adf but fails right at the end trying to zip up the files:

python3 ../inflate/pack_exe.py AmigaTestKit.exe AmigaTestKit.exe.tmpTraceback (most recent call last):
  File "../inflate/pack_exe.py", line 347, in <module>
    main(sys.argv)
  File "../inflate/pack_exe.py", line 295, in main
    (in_sz, out_sz) = process(open(argv[1], 'rb'), open(argv[2], 'wb'))
  File "../inflate/pack_exe.py", line 262, in process
    process_hunk(f, i)
  File "../inflate/pack_exe.py", line 157, in process_hunk
    packed = get_code('depacker_entry') + packed
  File "../inflate/pack_exe.py", line 89, in get_code
    assert id == HUNK_HEADER and x == 0
AssertionError
make: *** [Makefile:47: AmigaTestKit.exe] Error 1

[keirf]

I'm using binutils 2.34 but I don't suppose it should matter.

[keirf]

Actually it's perturbing because the Amiga hunk header which pack_exe.py is barfing on is actually created by another local Python script (mk_amiga_exe.py) immediately before. So this is really really weird. It can't be anything to do with the GNU toolchain because by this point the Elf exe has been stripped to a bare binary, put into an Amiga Hunk overcoat, and then passed to pack_exe.py.

[keirf]

Could you zip up AmigaTestKit.exe and attach it here?

[rjnorthrow]

I've zipped up all the AmigaTestKit.* files just in case you want to look at them.

atk.zip

[keirf]

Works for me ;) You were testing tip of master I assume (a776d0bf9e99837e4cdfde4ddececbf26fe99153)?

I am running Python 3.8.2 by the way. Perhaps I've inadvertently made myself unportable to other Python3 versions. What version are you running in Docker?

[rjnorthrow]

Ha, typical! ;)

I specifically checked out a776d0b and re-tried it with the same result.

I am running Python 3.8.5 in Docker.

Shall I paste in the whole output rather than the specific error just in case you spot something?

[rjnorthrow]

/Amiga-Stuff/testkit # make -j4
CC build.o
LD AmigaTestKit.elf
OBJCOPY AmigaTestKit.bin
zopfli AmigaTestKit.bin
python3 ../base/mk_amiga_exe.py AmigaTestKit.bin AmigaTestKit.exe
python3 ../inflate/pack_exe.py AmigaTestKit.exe AmigaTestKit.exe.tmp
../inflate/degzip -H AmigaTestKit.bin.gz AmigaTestKit.bin.tmp
Compressed size: 40143; Original size: 74572
Nr lits: 28531, Nr lens: 6975, Av len: 6
Lit/Len Ratio: 4.090466
Nr Codes: 44233, Av bits: 6.231479, Longcodes Ratio: 0.157303
Decompression requires leeway of 1 bytes
python3 ../base/mk_adf.py bootblock.bin AmigaTestKit.bin.tmp AmigaTestKit.adf
 BB + Compressed = 1168 + 40125 = 41293
 Load - Compressed = 41472 - 40125 = 1347
 Alloc - Decompressed = 74752 - 74572 = 180
 == 8 Tracks ==
rm -f AmigaTestKit.bin.tmp
Traceback (most recent call last):
  File "../inflate/pack_exe.py", line 347, in <module>
    main(sys.argv)
  File "../inflate/pack_exe.py", line 295, in main
    (in_sz, out_sz) = process(open(argv[1], 'rb'), open(argv[2], 'wb'))
  File "../inflate/pack_exe.py", line 262, in process
    process_hunk(f, i)
  File "../inflate/pack_exe.py", line 157, in process_hunk
    packed = get_code('depacker_entry') + packed
  File "../inflate/pack_exe.py", line 89, in get_code
    assert id == HUNK_HEADER and x == 0
AssertionError
make: *** [Makefile:47: AmigaTestKit.exe] Error 1

[keirf]

What if you don't do a parallel make?

[rjnorthrow]

Same error :(

[keirf]

Can you paste the log for serialised make?

[rjnorthrow]

/Amiga-Stuff/testkit # make
CC build.o
LD AmigaTestKit.elf
OBJCOPY AmigaTestKit.bin
zopfli AmigaTestKit.bin
../inflate/degzip -H AmigaTestKit.bin.gz AmigaTestKit.bin.tmp
Compressed size: 40142; Original size: 74572
Nr lits: 28535, Nr lens: 6974, Av len: 6
Lit/Len Ratio: 4.091626
Nr Codes: 44256, Av bits: 6.228579, Longcodes Ratio: 0.157109
Decompression requires leeway of 1 bytes
python3 ../base/mk_adf.py bootblock.bin AmigaTestKit.bin.tmp AmigaTestKit.adf
 BB + Compressed = 1168 + 40124 = 41292
 Load - Compressed = 41472 - 40124 = 1348
 Alloc - Decompressed = 74756 - 74572 = 184
 == 8 Tracks ==
rm -f AmigaTestKit.bin.tmp
python3 ../base/mk_amiga_exe.py AmigaTestKit.bin AmigaTestKit.exe
python3 ../inflate/pack_exe.py AmigaTestKit.exe AmigaTestKit.exe.tmp
Traceback (most recent call last):
  File "../inflate/pack_exe.py", line 347, in <module>
    main(sys.argv)
  File "../inflate/pack_exe.py", line 295, in main
    (in_sz, out_sz) = process(open(argv[1], 'rb'), open(argv[2], 'wb'))
  File "../inflate/pack_exe.py", line 262, in process
    process_hunk(f, i)
  File "../inflate/pack_exe.py", line 157, in process_hunk
    packed = get_code('depacker_entry') + packed
  File "../inflate/pack_exe.py", line 89, in get_code
    assert id == HUNK_HEADER and x == 0
AssertionError
make: *** [Makefile:47: AmigaTestKit.exe] Error 1

[keirf]

Ah hang on, I misread the error. The offending binary is actually inflate/depacker_entry. This is generated by bebbo's amiga-gcc toolchain. What version are you using of that? My 'm68k-amigaos-gcc --version' is 6.4.1b

[rjnorthrow]

Ah, I did make PREFIX=m68k-unknown-elf- for the inflate stuff. I guess I need to revisit the m68k-toolchain?

[keirf]

Yes sorry. You need https://github.com/bebbo/amiga-gcc -- it should be straightforward (LOL ;) ).

Confusingly there are two toolchains being used: m68k-elf-unknown and m68k-amigaos. You must use the latter where it is needed. That's what bebbo's repo builds for you.

[keirf]

And yeah I guess you could build both into your m68k-toolchain image. Makes sense.

[rjnorthrow]

Gotcha. Give me a while! There's no straightforward version for Alpine OS but I should be able to work it out.

[rjnorthrow]

Yep, will do

[rjnorthrow]

I've switched to using Ubuntu:20.04 as Bebbo has a handy script to download and install the necessary dependencies that works well on that OS. I was fighting myself trying to get it to work on Alpine. I have pushed the latest versions of m68k-toolchain and atk to github. They will start building on hub.docker.com automatically and will be available in a few hours (I've removed the default -j4 so that will slow it down). I have tested it locally and it's working fine now :)

[keirf]

Thanks for taking the time to sort it out!

[rjnorthrow]

No worries! I've just cleared everything down in my environment and pulled the images from Docker Hub and it's not working quite right as it's not copying the created zip file into the host machine from the docker image, so I'll work on that tomorrow.

[rjnorthrow]

Actually, it was fine, I was just missing a colon. If you're reading this MatrixCat3D, please try again now:

docker run -e MAKE_OPTS=-j4 -v $(pwd):/output --rm -ti rjnorthrow/atk

[keirf]

Tried it. Works great! Thanks again.

[MatrixCat3D]

Yep. Works perfectly. Thanks for sorting it out @rjnorthrow !