search

Halium / projectmanagement

143 stars 32 forks source link

[device-port] [suzu] Sony Xperia X #98

Open fredldotme opened 6 years ago

fredldotme commented 6 years ago

Tree: halium-7.1

The EGL_PLATFORM=hwcomposer environment variable is required for test_hwcomposer to succeed.

Kernel Support

This Halium abstraction uses a 4.4 kernel provided by Sony and CodeAurora, thus has the benefit of taking advantage of upstream kernel patches and follows Ubuntu's Xenial kernel series for AppArmor updates. This opens the door for potentially decade-long support of kernel patches. The Halium suzu device platform is determined to support Snap packages and is well positioned to do so long term.

Areas of improvements

Usability:

Fix Android init service start issues related to:

Possible improvements through use of FLOSS-compliant components

Upstreaming:

Build instructions

Build tests (even if you don't have the device) are welcome! The resulting image is arm64 and armhf multilib compatible.

Tested on Ubuntu 18.04 inside an LXD container:

# Install build prerequisites for Halium
PROJECT_ROOT=$HOME/Projects/Halium-7.1
DEVICE=suzu
BREAKFAST_DEVICE=aosp_f5121-userdebug
HALIUM_VER=halium-7.1

export LC_ALL=C

sudo dpkg --add-architecture i386
sudo apt update
sudo apt install git gnupg flex bison gperf build-essential \
  zip bzr curl libc6-dev libncurses5-dev:i386 x11proto-core-dev \
  libx11-dev:i386 libreadline6-dev:i386 libgl1-mesa-glx:i386 \
  libgl1-mesa-dev g++-multilib mingw-w64-i686-dev tofrodos \
  python-markdown libxml2-utils xsltproc zlib1g-dev:i386 schedtool \
  repo liblz4-tool bc lzop wget libssl-dev imagemagick

# Fetch & build
mkdir -p $PROJECT_ROOT
cd $PROJECT_ROOT

repo init -u https://github.com/Halium/android -b $HALIUM_VER --depth=1
repo sync

# Setup local_manifest repo xml
$PROJECT_ROOT/halium/devices/setup $DEVICE
repo sync --force-sync

# Optional workaround
# For some reason, sometimes the environment doesn't pick up the prebuild compiler.
# TL;DR: Try this if building fails
#
# Setup build environment using prebuilt compiler (uncomment exports to use the workaround):
# export CC=$PROJECT_ROOT/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8/bin/x86_64-linux-gcc
# export CXX=$PROJECT_ROOT/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8/bin/x86_64-linux-g++
# export LD=$PROJECT_ROOT/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8/bin/x86_64-linux-ld
# export CXX_INCLUDE_PATH=$PROJECT_ROOT/prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.11-4.8/sysroot/usr/include
# export C_INCLUDE_PATH=$CXX_INCLUDE_PATH

# Boot and system image creation
source build/envsetup.sh
breakfast $BREAKFAST_DEVICE
mka halium-boot && mka systemimage

Flashing images

Prerequisites:

The OEM binaries need to be flashed onto the oem partition:

sudo apt install unzip android-tools-fastboot || true
unzip SW_binaries_for_Xperia_AOSP_N_MR1_5.7_r1_v08_loire.zip
if [ -f SW_binaries_for_Xperia_AOSP_N_MR1_5.7_r1_v08_loire.img ]; then
fastboot flash oem SW_binaries_for_Xperia_AOSP_N_MR1_5.7_r1_v08_loire.img
fi

Flashing boot and system images: ~Edit 2018-09-13: switch to xenial-edge UBPorts rootfs~ Edit 2019-05-16: switch to xenial (devel) UBPorts rootfs

PROJECT_ROOT=$HOME/Projects/Halium-7.1
HALIUM_INSTALL_ROOT=$HOME/Projects/Halium-7.1/halium/halium-install
DEVICE=suzu

rm -r $HALIUM_INSTALL_ROOT || true
git clone https://github.com/JBBgameich/halium-install.git $HALIUM_INSTALL_ROOT

# Fetch latest successful armhf rootfs build
cd $PROJECT_ROOT
wget https://ci.ubports.com/job/xenial-rootfs-armhf/lastSuccessfulBuild/artifact/out/ubports-touch.rootfs-xenial-armhf.tar.gz

# At this point the device is expected to be in recovery mode
$HALIUM_INSTALL_ROOT/halium-install -p ut $PROJECT_ROOT/ubports-touch.rootfs-xenial-armhf.tar.gz $PROJECT_ROOT/out/target/product/$DEVICE/system.img
adb reboot bootloader
fastboot flash boot $PROJECT_ROOT/out/target/product/$DEVICE/halium-boot.img
fastboot reboot

Post-installation steps

The most current xenial rootfs (devel) build from ci.ubports.com is used in the following steps (2019-05-16). This shouldn't be needed anymore, keeping for reference.

# Mount rootfs read-write
sudo mount -o remount,rw /

sudo apt install pulseaudio-modules-droid-24

sync && sudo mount -o remount,ro / && sudo reboot
fredldotme commented 6 years ago

If you try a build by yourself and encounter problems, please specify your build environment (GNU/Linux distribution, changing default compiler, etc.) to help track down issues.

mdehollander commented 6 years ago

All right. I gave it a try. Thanks for providing the instructions. I got until the part building the systemimage, failing on something with checkpolicy on selinux.

More details:

Could the error I get be related to https://github.com/Halium/android_build/pull/4#issuecomment-421434682?

fredldotme commented 6 years ago

@mdehollander To get past the checkpolicy error you need to use the following sepolicy repo:

https://github.com/beidl/android_system_sepolicy/tree/halium-7.1

~You can remove-project the original and pull in my version within the device-specific repo manifest.~ EDIT: Now in the upstream PR.

fredldotme commented 6 years ago

A repo sync is suggested now to gain proper AppArmor/click & snap support.

Still missing:

Using systemd as the init manager and avoiding running into trouble whlie running /init on the lxc-confg-android container might require workarounds suggested in these cgmanager and lxc issues: https://github.com/lxc/cgmanager/issues/32 https://github.com/lxc/lxc/issues/1554

Specifically, the upstream lxc-android-config service shouldn't require cgmanager anymore.

mdehollander commented 6 years ago

I guess this is related to the not yet working audio. I get this error when making the systemimage:

ninja: error: '/home/mattias/Projects/Halium-7.1/out/target/product/suzu/obj_arm/SHARED_LIBRARIES/libaudiopolicymanager_intermediates/export_includes', needed by '/home/mattias/Projects/Halium-7.1/out/target/product/suzu/obj_arm/SHARED_LIBRARIES/libaudiopolicyservice_intermediates/import_includes', missing and no known rule to make it

Setting USE_CUSTOM_AUDIO_POLICY to 0 in device/sony/common/CommonConfig.mk brings me to:

ninja: error: 'out/target/product/suzu/symbols/system/vendor/lib/librmnetctl.so', needed by '/home/mattias/Projects/Halium-7.1/out/target/product/suzu/system/vendor/lib/librmnetctl.so', missing and no known rule to make it

Is there a part I did not configure correctly?

z3ntu commented 6 years ago

@mdehollander Looks like you don't have the vendor repo added (or the vendor repo doesn't contain librmnetctl.so).

fredldotme commented 6 years ago

I guess this is related to the not yet working audio. I get this error when making the systemimage:

ninja: error: '/home/mattias/Projects/Halium-7.1/out/target/product/suzu/obj_arm/SHARED_LIBRARIES/libaudiopolicymanager_intermediates/export_includes', needed by '/home/mattias/Projects/Halium-7.1/out/target/product/suzu/obj_arm/SHARED_LIBRARIES/libaudiopolicyservice_intermediates/import_includes', missing and no known rule to make it

Setting USE_CUSTOM_AUDIO_POLICY to 0 in device/sony/common/CommonConfig.mk brings me to:

Interesting, I'll try a test with that flag, hasn't happened for me yet.

ninja: error: 'out/target/product/suzu/symbols/system/vendor/lib/librmnetctl.so', needed by '/home/mattias/Projects/Halium-7.1/out/target/product/suzu/system/vendor/lib/librmnetctl.so', missing and no known rule to make it

Is there a part I did not configure correctly?

You could comment out the manual copying of librmnetctl.so files to the system image (within device/sony/loire/platform.mk) and uncomment them later in case the PRODUCT_PACKAGES additions aren't enough.

mdehollander commented 6 years ago

Thanks @beidl. Commenting out the copying worked for me. At least it is compiling and now at 50%.

fredldotme commented 6 years ago

Thanks @beidl. Commenting out the copying worked for me. At least it is compiling and now at 50%.

Great! Have you tried flashing the image onto your Xperia X? There's a chance it might require an additional apt repo which has already been removed.

mdehollander commented 6 years ago

I tried flashing the image to my device, but had to figure out how to do that. And I think I did not do it correctly since I got this error when running halium-install:

I: Unmounting images
I: Shrinking images
I: Pushing rootfs and android image to /data via ADB
error: device not found
Error: Couldn't copy the files to the device, is it connected?
I: Cleaning up host
I: Cleaning up device
error: device not found

I looked a bit at http://docs.halium.org/en/latest/supplementary/devices/nicki.html to find out how to get into fastboot and recovery mode. However, in recovery mode (green led), the device was not found with adb. Therefore I booted into TWRP recovery and then connected the device to the laptop to start halium-install. How did this work for you?

fredldotme commented 6 years ago

@mdehollander it could be possible you're missing the udev device rules to establish adb and fastboot connections. You could try installing them using your distributions package manager or find a udev rules file on GitHub.

fredldotme commented 6 years ago

Feel free to test a prebuilt image in case you cannot get the build up and running yourself:

https://drive.google.com/open?id=1OJjTosp7jFaYhEYYkB9Mxf3lA0HPIXY6

https://forums.ubports.com/topic/1669/ubuntu-touch-on-sony-xperia-x

retro486 commented 5 years ago

The prebuilts don't seem to boot for me unless I'm missing a step (flash something else first?). I did flash the nougat OEM blob first. I also can't seem to get the build working but I'll need to try again to get the specific error as it was from a month ago... It would be great to see more Sony Open devices since Sony has been keeping the program up and releasing updated binaries and kernels.

fredldotme commented 5 years ago

@retro486 how far along can the bootup process get? Are you able to get an active USB networking connection? You could check using dmesg if a networking device or similar shows up.

retro486 commented 5 years ago

After booting TWRP and mounting the rootfs.img and changing the ssh init script to run at runlevel 1 (in addition to 2-5) I was able to ssh in. I'm seeing a lot of crashing services in dmesg but I haven't tried the extra sound in the guide above yet... This is using the latest UBports rootfs.

Edit: but it's booting which is awesome!

Edit 2: Woo! After your post-installation fixes listed above I've got Ubuntu Touch on my Xperia X. Holy smokes that's awesome!

Edit 3: I still had to perform a strange fix in that after using halium-install-standalone.sh I booted into TWRP, mounted rootfs.img and changed /etc/init/ssh.conf:

start on runlevel [2345] stop on runlevel [!2345]

to:

start on runlevel [12345] stop on runlevel [!12345]

I confirmed I was booting into runlevel 2 so I don't know why I had to add runlevel 1...

retro486 commented 5 years ago

Final comment then I'll move this to the UT forums but I'm using the stock UT rootfs tarball from their custom Halium wiki. I couldn't get the halium reference rootfs to open SSH.

fredldotme commented 5 years ago

Tbf there's still an issue with attaching to an already running LXC container, seems to be a kernel-side issue. Among other things this seems to be the root cause of Plasma Mobile refusing to successfully boot into a graphical session.

mauricioduarte01 commented 5 years ago

Can't wait to try this. I am looking to buy a new phone and I think this is the best candidate. Thank you for the hard work!

xsetiadi commented 5 years ago

how about anbox? can this port run anbox?

fredldotme commented 5 years ago

The kernel has yet to be modified to run Anbox but in theory, as soon as that's worked out, it should run just fine.

xsetiadi commented 5 years ago

sony Xperia x and x compact is had similar hardware, so will this port can run on Xperia x compact?

fredldotme commented 5 years ago

The Xperia X and Xperia X Compact are different devices, you might want to follow this project management issue instead: https://github.com/Halium/projectmanagement/issues/103

xsetiadi commented 5 years ago

ok, I think it's just can run on Xperia x compact like G7 has done to make compatibility layer to run sailfish x on Xperia x compact

fredldotme commented 5 years ago

I have enabled the kernel to support the Anbox-required additional Binder instance. As soon as the Xperia X hits the list of supported Anbox devices we can mark Anbox support as fully available.

xsetiadi commented 5 years ago

wow, sounds good, whats next?

fredldotme commented 5 years ago

Missing pieces are audio while calling, camera and video playback.

xsetiadi commented 5 years ago

does this will work too for xperia x dual? F5122?

fredldotme commented 5 years ago

In theory yes, practically I don't have the device available. But since I modified the source in a way that's not specific to F5121 it should work just fine. The only area I could see issues with is ofono needing an override to support the second SIM tray.

fredldotme commented 5 years ago

Calling now works (with caveats). For the time being you need a test version of telepathy-ofono on your device to enable calling:

sudo ubports-qa install xenial_-_haliumfixup
xsetiadi commented 5 years ago

it's getting interesting by the time

fredldotme commented 5 years ago

A new build will be released soon with functional hardware video decoding & partial camera support (currently only pictures & no flash).

neutralinsomniac commented 5 years ago

Hey there. Huge props for getting this working. As far as I can tell, I followed the installation instructions here: https://forum.xda-developers.com/xperia-x/development/rom-ubuntu-touch-sony-xperia-x-t3958100, but boot is failing. Looks like my partitions aren't set up correctly?

Relevant portion of on-device dmesg follows:

[    2.464481] initrd: checking filesystem integrity for the userdata partition
[    2.476468] [SDFAT](mmcblk0p51[259:19]): trying to mount...
[    2.476911] [SDFAT](mmcblk0p51[259:19]): invalid boot record signature
[    2.477084] [SDFAT](mmcblk0p51[259:19]): failed to recognize fat type
[    2.477274] [SDFAT](mmcblk0p51[259:19]): failed to mount! (-22)
[    2.478840] mount: mounting /dev/mmcblk0p51 on /tmpmnt failed: Invalid argument
[    2.481495] umount: can't umount /tmpmnt: Invalid argument
[    2.511722] initrd: checking filesystem for userdata took (including e2fsck) 0 seconds
[    2.523541] dumpe2fs 1.43.4 (31-Jan-2017)
[    2.525348] dumpe2fs: Bad magic number in super-block while trying to open /dev/mmcblk0p51
[    2.550518] resize2fs 1.43.4 (31-Jan-2017)
[    2.552663] resize2fs: Bad magic number in super-block while trying to open /dev/mmcblk0p51
[    2.553558] Couldn't find valid filesystem superblock.
[    2.554543] initrd: resized userdata filesystem to fill /dev/mmcblk0p51
[    2.554828] initrd: mounting /dev/mmcblk0p51
[    2.576616] sh: ext4: unknown operand
[    2.584990] [SDFAT](mmcblk0p51[259:19]): trying to mount...
[    2.585502] [SDFAT](mmcblk0p51[259:19]): invalid boot record signature
[    2.585625] [SDFAT](mmcblk0p51[259:19]): failed to recognize fat type
[    2.585810] [SDFAT](mmcblk0p51[259:19]): failed to mount! (-22)
[    2.587133] mount: mounting /dev/mmcblk0p51 on /tmpmnt failed: Invalid argument
[    2.594458] initrd: boot mode: halium
[    2.594751] initrd: Halium rootfs is  
[    2.596141] initrd: mounting system rootfs at /halium-system
[    2.598525] initrd: WARNING: Android system image not found.
[    2.598861] initrd: mounting   (user mode)
[    2.600841] mount: can't find /halium-system in /proc/mounts
[    2.601674] initrd: mounting android system image (/tmpmnt/android-rootfs.img) ro, in /android-unknown (unknown mode)
[    2.601987] initrd: mounting android system image from system rootfs
[    2.605957] mount: mounting /halium-system/var/lib/lxc/android/android-rootfs.img on /android-unknown failed: No such file or directory
[    2.606709] initrd: WARNING: Failed to mount Android system.img.
[    2.607939] initrd: Normal boot
[    2.609803] mount: mounting /halium-system on /root failed: Invalid argument
[    2.613569] mount: mounting /tmpmnt on /root/userdata failed: Invalid argument
[    2.615610] mount: mounting /android-rootfs on /root/var/lib/lxc/android/rootfs failed: No such file or directory
[    2.627112] grep: /android-system/build.prop: No such file or directory
[    2.628703] initrd: WARNING: Didn't find a device name. Is the Android system image mounted correctly?
[    2.628988] initrd: device is unknown
[    2.629295] initrd: This rootfs does not have any writable-paths defined
[    2.629572] initrd: checking fstab /root/var/lib/lxc/android/rootfs/fstab* for additional mount points
[    2.634693] cat: can't open '/root/var/lib/lxc/android/rootfs/fstab*': No such file or directory
[    2.636910] initrd: moving Android system to /android/system
[    2.638881] mount: mounting /android-system on /root/android/system failed: Invalid argument
fredldotme commented 5 years ago

Hello @neutralinsomniac! Have you tried wiping everything, including /data & /system? Also, does halium-install show any errors, like when it's sending the final image files to the device?

EDIT: Also, are you using f2fs? The system expects the partition to be formatted with ext4.

neutralinsomniac commented 5 years ago

Hello @neutralinsomniac! Have you tried wiping everything, including /data & /system?

Looks like this was the ticket. I was coming from a fresh restore of Android 8. Went into TWRP, formatted /data and /system, re-did all the installation steps, and now it looks like ubuntu touch is booting properly! Thank you! Not sure if it would be helpful to add those steps to the install guide?

fredldotme commented 5 years ago

Hello @neutralinsomniac! Have you tried wiping everything, including /data & /system?

Looks like this was the ticket. I was coming from a fresh restore of Android 8. Went into TWRP, formatted /data and /system, re-did all the installation steps, and now it looks like ubuntu touch is booting properly! Thank you! Not sure if it would be helpful to add those steps to the install guide?

Makes sense, I'll include it in the XDA forum post. Thanks for testing!

retro486 commented 4 years ago

Thanks for this! I just wanted to add that in order to get OTAs to install (if desired) people need to do this additional step:

fastboot flash recovery $PROJECT_ROOT/out/target/product/$DEVICE/recovery.img

This will replace TWRP (presumably if they flashed it and didn't just fastboot boot it) with the Ubuntu Touch recovery allowing installs of downloaded OTAs.

slowcyclist commented 4 years ago

Hi all, When running test_sensors with no arguments, it lists 29 sensors, but if I run it with a sensor index as argument, I actually get sensor data only for sensor #0 (accelerometer), which is odd. In particular, I wonder why I get no data from the magnetometer since it was working in Sailfish (I did not run test_sensors in Sailfish, though). Does anyone have an idea of what could be wrong?

fredldotme commented 4 years ago

I believe Sailfish is using sensorfw as sensor middleware, whereas with UT on suzu we're still using the old Canonical way of doing things through platform-api.

We're using sensorfw for Halium 9 based ports already, but it's conflicting with platform-api. Therefore Halium 9 based ports use a slightly modified variant of the rootfs. I'm not sure how much work it would be to make the standard rootfs use sensorfw, especially since it would mean dropping some core devices which don't have open source device trees available.

So currently, if you want to use a magnetometer, you're probably better off making use of a Halium 9 based adaptation of UT.

slowcyclist commented 4 years ago

Hmm. The magnetometer sensor is listed as "AK09915", and when I looked at sensorfw code it seemed to me those natively implemented are ak897X. So, I was guessing that Sailfish was getting the Xperia X magnetometer data using sensorfw's hybrismagnetometeradaptor. Indeed libhybris sensors are not far to fully work in your port since many devices are identified. Using strace on test_sensor, it seems that polling the sensors simply does not return any data (except for sensor 0, the accelerometer). One difference between Sailfish and your port, is that Sailfish is using an older version of android as base for that phone; I was suspecting this could explain the difference and hoping it could be simple to fix.

Now, I did not take a look at platform-api and maybe you mean platform-api is not accessing the android drivers through libhybris, and it's that api code that needs to be fixed for getting magnetometer data?

I probably won't be able to investigate Halium 9 adaptation on that phone, because it's my only phone and I need it in a usable state. Yet, out of curosity, you believe that adaptation is feasible?

fredldotme commented 4 years ago

For the magnetometer to work on UT you'd either have to implement the feature in what we have already, or make it possible to run sensorfw on the Xperia X + all core devices that UBports supports. I'm not sure though if supporting all core devices using sensorfw would even be possible as we don't have access to some of the device trees (they were proprietary when Canonical worked with those vendors).

fredldotme commented 4 years ago

@slowcyclist Seems magnetometer sensor support was in development but never merged in the Canonical days. We're reviving it: https://github.com/ubports/platform-api/pull/8

Beryesa commented 1 year ago

Any updates?

slowcyclist commented 1 year ago

Yes, most sensors now work on the xperia x in UT (still xenial for me, at this point): gps, magnetometer, acceleration, gyro, pressure, light ... Temperature gives no indication, but maybe there is just no thermometer in the device.