Deb file with long path fails to install sometimes

tyilo commented 6 years ago

What are the steps to reproduce this issue?

Create a new theos project:

$ nic.pl
NIC 2.0 - New Instance Creator
------------------------------
[1.] iphone/activator_event
[2.] iphone/application_modern
[3.] iphone/cydget
[4.] iphone/flipswitch_switch
[5.] iphone/framework
[6.] iphone/ios7_notification_center_widget
[7.] iphone/library
[8.] iphone/notification_center_widget
[9.] iphone/preference_bundle_modern
[10.] iphone/tool
[11.] iphone/tweak
[12.] iphone/xpc_service
Choose a Template (required): 10
Project Name (required): foobar
Package Name [com.yourcompany.foobar]: 
Author/Maintainer Name [tyilo]: 
Instantiating iphone/tool in foobar/...
Done.

Make the following directory in layout and add two specific files:

cd foobar
DIR="layout/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM"
mkdir -p "$DIR"
cd "$DIR"
curl https://pomf.pyonpyon.moe/osjisf.swf -o library.swf
curl https://pomf.pyonpyon.moe/jinfnl.xml -o platform.xml

Run make package install multiple times (like 10 times)

What happens?

Some of the installations succeeds and some fails. When it fails, the error message differs.

Here are some outputs that I have observed: https://gist.github.com/Tyilo/44360831d14087821365782b0a4c4e0d

What were you expecting to happen?

Installation should always succeed.

Any logs, error output, etc?

See above.

Any other comments?

Actually the problem seems to be with dpkg on my iPhone: Just repeatedly running dpkg -i on the generated deb will succeed sometimes and fail sometimes. I didn't have this problem before the switch to using dm.pl.

I'm using iOS 9.3.3 and the installed dpkg version is 1.18.10. Maybe dpkg needs to be updated for Cydia?

What versions of software are you using?

Operating System: Linux

Toolchain Version:

iOS Toolchain for Linux

LLVM + Clang 3.9.1
LD64 264.3.102
CCTools 877.5

Built by CoolStar w/ <3

SDK Version: iPhoneOS9.3.sdk

kirb commented 6 years ago

This is really weird. Looks like maybe the Archive::Tar module is having a bad day generating a corrupt tar, or worse, those particular files are triggering some sort of buffer overflow in dpkg or tar during installation. I’ve managed to repro with dm.pl (thanks for the test case!) but haven’t looked at how dpkg-deb behaves yet, so not sure yet whether this is a bug in packaging or installation.

tyilo commented 6 years ago

Putting

_THEOS_PLATFORM_DPKG_DEB_COMPRESSION = gzip

in the top of my Makefile fixes this, although it would of course be better if lzma also worked.

tyilo commented 6 years ago

Just using gzip compression doesn't always seem to work either :(

kirb commented 6 years ago

Could you try the current master commit of dm.pl? Use this:

cd $THEOS/vendor/dm.pl
git fetch origin master
git checkout master

tyilo commented 6 years ago

Nope, still doesn't work

tyilo commented 6 years ago

Here is a deb file generated (but using Architecture: amd64) using the above method:

foobar.deb (sha256sum: 1fd81203cea8e6ee4db41f15977b2c276b6688e9b2b194e0824227a144324226)

I'm using dpkg version 1.18.24 on Arch Linux to do the following tests using the above file. They show that something is clearly wrong:

First we download the file:

$ curl https://my.mixtape.moe/vazmjt.deb -o foobar.deb
$ sha256sum foobar.deb 
1fd81203cea8e6ee4db41f15977b2c276b6688e9b2b194e0824227a144324226  foobar.deb

The output from dpkg --contents look normal:

$ dpkg --contents foobar.deb
drwxr-xr-x root/wheel        0 2018-06-30 20:59 .
drwxr-xr-x root/wheel        0 2018-06-30 20:59 ./usr
drwxr-xr-x root/wheel        0 2018-06-30 20:59 ./usr/bin
-rwxr-xr-x root/wheel   131856 2018-06-30 20:59 ./usr/bin/foobar
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE
drwxr-xr-x root/wheel        0 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM
-rw-r--r-- root/wheel      237 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM/platform.xml
-rw-r--r-- root/wheel     5014 2018-06-30 20:54 ./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM/library.swf

dpkg-deb --extract also seems to work:

$ dpkg-deb --extract foobar.deb .
$ find .
.
./var
./var/mobile
./var/mobile/FooBarTweak
./var/mobile/FooBarTweak/FooBarOriginal.app
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM/library.swf
./var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM/platform.xml
./usr
./usr/bin
./usr/bin/foobar
./foobar.deb

However if we install the package with dpkg and then list the files installed, we can see that something is wrong. The filenames of library.swf and platform.xml has been appended with garbage:

$ sudo dpkg --install foobar.deb                                                                                                                                                            
Selecting previously unselected package com.yourcompany.foobar.
(Reading database ... 0 files and directories currently installed.)
Preparing to unpack foobar.deb ...
Unpacking com.yourcompany.foobar (0.0.1-3+debug) ...
Setting up com.yourcompany.foobar (0.0.1-3+debug) ...
$ dpkg --listfiles com.yourcompany.foobar
/.
/usr
/usr/bin
/usr/bin/foobar
/var
/var/mobile
/var/mobile/FooBarTweak
/var/mobile/FooBarTweak/FooBarOriginal.app
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM/platform.xmlumble!M
/var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM/library.swflumblea

This also matches the installed files:

$ ls /var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM/
 library.swflumblea  'platform.xmlumble!M'$'\001'

If we now try to reinstall the package we also get an error:

$ sudo dpkg --install foobar.deb 
(Reading database ... 17 files and directories currently installed.)
Preparing to unpack foobar.deb ...
Unpacking com.yourcompany.foobar (0.0.1-3+debug) over (0.0.1-3+debug) ...
dpkg: error processing archive foobar.deb (--install):
 newline not allowed in archive object name './var/mobile/FooBarTweak/FooBarOriginal.app/META-INF/AIR/extensions/com.distriqt.FacebookUtils/META-INF/ANE/iPhone-ARM/platform.xml:20:
q'
Errors were encountered while processing:
 foobar.deb

However uninstalling seems to work:

$ sudo dpkg --remove com.yourcompany.foobar 
(Reading database ... 17 files and directories currently installed.)
Removing com.yourcompany.foobar (0.0.1-3+debug) ...
dpkg: warning: while removing com.yourcompany.foobar, directory '/var' not empty so not removed
dpkg: warning: while removing com.yourcompany.foobar, directory '/usr/bin' not empty so not removed

tyilo commented 6 years ago

Here is another way to reproduce the issue (requiring no downloads of library.swf or platform.xml):

Put the following in Dockerfile and build it (sudo docker build .):

FROM debian

RUN apt-get update && apt-get install -y git xxd

RUN git clone https://github.com/theos/dm.pl.git
WORKDIR dm.pl

ENV LONGFILE aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

RUN mkdir -p /layout/DEBIAN
RUN printf "Package: com.example.deb-test\nName: deb test\nVersion: 0.0.1\nArchitecture: amd64\nDescription: deb test\nMaintainer: Foo <foo@example.com>\n" > /layout/DEBIAN/control
RUN touch /layout/$LONGFILE

RUN dpkg-deb --build /layout test-dpkg.deb
RUN dpkg --contents test-dpkg.deb
RUN dpkg --install test-dpkg.deb
RUN dpkg --listfiles com.example.deb-test | xxd
RUN ls -l /$LONGFILE

RUN dpkg --remove com.example.deb-test

RUN ./dm.pl /layout test-dm.deb
RUN dpkg --contents test-dm.deb
RUN dpkg --install test-dm.deb
RUN dpkg --listfiles com.example.deb-test | xxd
RUN ls -l /$LONGFILE

ENTRYPOINT ["/bin/true"]

The output should be:

$ sudo docker build .
...
Step 18/20 : RUN dpkg --listfiles com.example.deb-test | xxd
 ---> Running in b5227210292d
00000000: 2f2e 0a2f 6161 6161 6161 6161 6161 6161  /../aaaaaaaaaaaa
00000010: 6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
00000020: 6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
00000030: 6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
00000040: 6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
00000050: 6161 6161 6161 6161 6161 6161 6161 6161  aaaaaaaaaaaaaaaa
00000060: 6161 6161 6161 6161 ffff 710a            aaaaaaaa..q.
Removing intermediate container b5227210292d
 ---> 3edaa105c83e
Step 19/20 : RUN ls -l /$LONGFILE
 ---> Running in 36e53d197097
ls: cannot access '/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa': No such file or directory
The command '/bin/sh -c ls -l /$LONGFILE' returned a non-zero code: 2

As we can see when the deb is created using dpkg-deb there is no problem with installing the deb, however when using dm.pl it somehow corrupts the filename when installing by adding garbage to the end of the filename.

theos / dm.pl