search

efidroid / projectmanagement

EFIDroid Project Management
118 stars 12 forks source link

efidroid on Android 9 and 10 devices ? #152

Open aiamuzz opened 4 years ago

aiamuzz commented 4 years ago

Hi,

I just want to know if efidroid supports devices with 6 GB RAM and 64/128 GB Storage devices running Android 9 and Android 10 ?

thanks.

feherneoh commented 4 years ago

Last Android release that was properly supported was 6, and 7 did work in some cases, all only on 32bit devices, so no, they aren't supported at all

aiamuzz commented 4 years ago

Last Android release that was properly supported was 6, and 7 did work in some cases, all only on 32bit devices, so no, they aren't supported at all

its sad ... all these multi boot solutions were a little too early(they were implemented when devices and their hardware were weak(especially RAM) and insufficient(storage) ... But, when the time is perfect, for such multi boot solutions ... when we have devices with ample hardware in terms of RAM(memory) and Storage(64/128) these solutions are going extinct !!!

there were atleast 4 projects that were available ...

all of the above have pretty much faded as far as the latest Android(versions 9 upwards)

If there was ever a time for multiboot concepts ... its RIGHT NOW ... for devices with sufficiently decked up hardware !!!

feherneoh commented 4 years ago

Problem is, Google changes stuff way too much with every release I don't know safestrap, MultiROM has multiple methods all of which do perma-patch the system, so does Multi-Boot patch as the name implies The Magisk way could work, I assume that would only multiboot Android, not the kernel.

M1cha commented 4 years ago

@aiamuzz you're 100% right. What many people don't know is that I started working on multiboot solutions in 2012. Which means (excluding 2019) I've been working on that for 7 years. EFIDroid alone started in 2015 which means I worked on that one for 4 years.

The reason why I stopped is that the amount of work that needed to be done increased everytime I found a way to solve an issue. That means the more work I had done, the further away we were from the next stable release.

The reason for that is that I always tried to be compatible(or have the technical potential to be) with all devices while requiring not a single multiboot-supporting patch from the operating system (Android, even potentially supporting non-Android OSs).

So I started replacing/extending the bootloader stage so we don't need patched kernels with kexec or whatever and have a clean, unified starting point. Intially this was done using LK, then LK-ubootapi-GRUB, then LK-UEFIAPI-GRUB, then LK-UEFI, then pure UEFI(If I'd still be working on it, it'd be linuxboot with support for an optional UEFI runtime).

The next step is actually booting the kernel which on smartphones is a very complicated process involving vendor-specific devicetree patches with information you have to be able to obtain somehow. But I think we solved that problem since a ton of time went into that.

Now the linux kernel is running and the non-multiboot aware operating system has to be tricked into preparing the filesystem somehow. Naturally I started doing the same that everybody does: patching the fstab. Since not everything in Android is using fstab though(especially on qcom devices) you have to start looking through the whole ramdisk patching every occurence. Eventually you find out that some paths are hardcoded in apks/jars on the system partition and there goes your generic multiboot without ROM support. I'm not even getting into selinux support here.

So while every other multiboot solution stopped there, I actually found another way which is actually exactly the same which I did in my very first multiboot solution for Motorola Defy but would still not need any OS modifications: intercepting the communication between kernel and userspace(which efidroid already did in a slow and buggy way for allowing you to use any unmodified recovery which was also unheard of before). Basically, UEFI would scan the mostly static, handwritten assembly startup-code of the linux kernel and insert code which is alive while the kernel is still running, handling all the syscalls we're interested in(mostly mounts and file IO). Once fully implemented(which I never did) the cat and mouse game would be reduced to supporting kernels, not ROMs which would make this the most compatible solution on earth.

So while all this is a ton of work already, unfortunately it doesn't stop there because of EFIDroid's second goal: providing the best user experience imaginable. Which basically means we need a full-featured UI(Material design, animation, icons, fonts, ...) with support for touch screens. So while we had the UI part 100% solved by porting QT5 to UEFI(also previously deemed impossible, and now QT is even doing something similar officially), the driver part was a problem. It extends to more basic things too though wich means we'd need to solve this even without a fancy UI: LCD, MMC, sdcard, USB and all the peripherals required by these. That's why we used LK as the driver back bone for UEFI. but then we found out that LK drivers are buggy and incomplete and that qcom would stop supporting them migrating to closed source uefi drivers. Early in this wall of text(sorry) I mentioned linuxboot with support for an optional UEFI runtime. This is were that comes in. Basically we'd use a mainline linux kernel, add very basic driver support(like we did in UEFI/LK) but would have an easier time doing so because we'd not have to emulate a ton of linux APIs and remove all threading and MMU functionality. linuxboot is another project which does that for x86 only that they don't have to go minimal since x86 supports full re-initialization of the hardware.

So to summarize, there are two ways to do multiboot:

Even though we brought together over 300 people on slack(which is insane btw and I'm really thankful for that) I've never found somebody who has the knowledge and the motivation to put in the same amount work as I did working on the heart of EFIDroid and not just doing device ports. I don't want to downplay the huge contributions of some people though. Thanks @feherneoh and all the people that I forgot(which are probably quite a few).

To give this some personal background as well: you can imagine that a person's interests can shift quite a lot within 7 years. some key points:

I'll definitely stick around answering questions (multiboot related or not) and you can be sure I read every single comment on github. But unless somebody with the skills dedicates themselves to contribute as much as I did to cut the amount of work on my side in half, double the motivation and turn me from THE main developer into a co-developer, I'm not seeing this project going anywhere.

aiamuzz commented 4 years ago

The Magisk way could work, I assume that would only multiboot Android, not the kernel.

Oh ... but i got a sorry reply from its creator (https://github.com/topjohnwu/Magisk/issues/2076#issuecomment-554270943)

But unless somebody with the skills dedicates itself to contribute as much as I did to cut the amount of work on my side in half, double the motivation and turn me from THE main developer into a co-developer, I'm not seeing this project going anywhere.

all i can do is 'Tip my Hat' to your hard work ... and thanks for 'thinking out loud' about your journey and hard work that has gone into the efidroid project !!! ... Phew ... i could literally visualize it !!!

I guess Google's to blame ... to be making it difficult each time they come up with a release !!!

  1. For increasing the difficulty in allowing/supporting multiboot feature.

  2. FOR HANDLING THE MEMORY(RAM) ... I am really really pissed and frustrated with Android on the way it handles RAM ... I am no techie but i have been a linux user for the last 8-10 years ... and if Linux and Android stem from the same pedigree(kernel) then i just hate Google for doing a shoddy job about handling RAM !!!

Look at Apple and their iPhone's ... they had ONLY 512 mb(as far as versions 4 & 4s) ... as recent as 2017 even their iPhone 8 with 256Gb Storage had just 2 GB of RAM ... and my device is a 2016 OnePlue 3T which has 6GB RAM ... and guess how much of available RAM it has at an average ... just about 2GB out of the massive 6GB ... i mean 'what the hell' !!!

My 2019 Linux Distro (Deepin) comes up with lesser and lesser memory footprint from its last iteration ... on boot my laptop has booted into a full blown OS with just 1GB of memory consumption ... and Android eats up 4GB of my 6GB ... seems to me that we just splurging away our hard earned money on google's inability to rein in the memory usage from iteration to iteration ... instead they are coming out with iterations that hog all of the memory resources ... (sorry for my 2nd point may have been off topic) but i really hate having paid for 6 GB and get only 2 GB !!!

Iphone 11 has 4 GB RAM ... and we have Android phones with 12 GB RAM ... What the Hell !!! JUST WISH SOMEBODY CAME UP WITH A BETTER WAY TO HANDLE MEMORY(RAM) ... AND I(THE USER) GETS THE MAXIMUM MEMORY(RAM) FOR THE BUCK HE PAID FOR !!!

No ... techie ... but is it even possible in theory to have a memory footprint as low as my full blown latest linus desktop OS ?

  • I don't have a need for multiboot anymore(the devices I own don't even have more than 1 ROM, so what would I boot into?)

I agree ... even i have my Linux daily OS besides the Windows(a necessary evil in case you have to run an app that exists only for this OS) ... even though it may not be used much and by many people but for the few that really use it ... its a definite boon ... for ROM freaks ... for ROM testers ... for Devs !!!

But unless somebody with the skills dedicates itself to contribute as much as I did to cut the amount of work on my side in half, double the motivation and turn me from THE main developer into a co-developer, I'm not seeing this project going anywhere.

I really hope ... i really do ... that things did get back on track for efidroid ... and ... most importantly google softened their approach and supports / allows such multiboot tech to thrive from one iteration to another !

And last but not the least @M1cha ... your efidroid(with all its UI and implemntation(process flows) ... is by far the most elegant implementation of a multiboot rig ... its sad that i came across it only now after having bought a higher spec'ed hardware device !!! but nevertheless its the most elegant of implementations ... i tip my hat once again to the efidroid team !!!

M1cha commented 4 years ago

First, thanks for your kind words :)

No ... techie ... but is it even possible in theory to have a memory footprint as low as my full blown latest linus desktop OS ?

While it is true, that Android is really inefficient and I wish that they'd finally drop the "we have to replace every opensource technology with our own version of it" attitude (and replace java with something more native and which does not support exceptions like rust), the RAM problem is not as big as it appears. Unlike Desktop-Linux, Android tries to use as much RAM as possible to speed up loading processes and resources which have been loaded before. So even if your RAM might appear full it's actually not because it's just a giant cache which can be dropped if it turns out the the new process needs 4GB RAM none of which can be reused from the cache.

aiamuzz commented 4 years ago

I really hope @topjohnwu ... and his Magisk team and your efidroid team could hit a breakthrough implementation of a single multiboot solution ... something like the Grub we use on Linux machines ...

I like the fact that Magisk a systemless implementation does not have to be adpated to each and every device ... just flash a single file across all devices ... a similar multiboot solution a single application(installer) that will apply to all android devices would be such a great option to have !!!

I really Hope ... for i am just human !!!

feherneoh commented 4 years ago

I really hope @topjohnwu ... and his Magisk team and your efidroid team could hit a breakthrough implementation of a single multiboot solution ... something like the Grub we use on Linux machines ...

I like the fact that Magisk a systemless implementation does not have to be adpated to each and every device ... just flash a single file across all devices ... a similar multiboot solution a single application(installer) that will apply to all android devices would be such a great option to have !!!

I really Hope ... for i am just human !!!

That's just impossible

aiamuzz commented 4 years ago

I really hope @topjohnwu ... and his Magisk team and your efidroid team could hit a breakthrough implementation of a single multiboot solution ... something like the Grub we use on Linux machines ... I like the fact that Magisk a systemless implementation does not have to be adpated to each and every device ... just flash a single file across all devices ... a similar multiboot solution a single application(installer) that will apply to all android devices would be such a great option to have !!! I really Hope ... for i am just human !!!

That's just impossible

I know i fear ... probably it may be possible if google decides to soften and allow it such tech !!!

feherneoh commented 4 years ago

I really hope @topjohnwu ... and his Magisk team and your efidroid team could hit a breakthrough implementation of a single multiboot solution ... something like the Grub we use on Linux machines ... I like the fact that Magisk a systemless implementation does not have to be adpated to each and every device ... just flash a single file across all devices ... a similar multiboot solution a single application(installer) that will apply to all android devices would be such a great option to have !!! I really Hope ... for i am just human !!!

That's just impossible

I know i fear ... probably it may be possible if google decides to soften and allow it such tech !!!

Still won't help, as for an usable dualboot that doesn't fry your device over time you need pre-kernel code, and there is just no way to make that completely universal. You MUST use builds customized for specific devices. Magisk can be universal, because anything not part of userspace it just pre-patches at install

aiamuzz commented 4 years ago

I really hope @topjohnwu ... and his Magisk team and your efidroid team could hit a breakthrough implementation of a single multiboot solution ... something like the Grub we use on Linux machines ... I like the fact that Magisk a systemless implementation does not have to be adpated to each and every device ... just flash a single file across all devices ... a similar multiboot solution a single application(installer) that will apply to all android devices would be such a great option to have !!! I really Hope ... for i am just human !!!

That's just impossible

I know i fear ... probably it may be possible if google decides to soften and allow it such tech !!!

Still won't help, as for an usable dualboot that doesn't fry your device over time you need pre-kernel code, and there is just no way to make that completely universal. You MUST use builds customized for specific devices. Magisk can be universal, because anything not part of userspace it just pre-patches at install

:( :( :( ... oh ok ... my worst of fears ... i am no techie ... but seems you people know what you are talking about ... and i totally understand what you are saying conceptually !!!

Iphone 11 has 4 GB RAM ... and we have Android phones with 12 GB RAM ... What the Hell !!! JUST WISH SOMEBODY CAME UP WITH A BETTER WAY TO HANDLE MEMORY(RAM) ... AND I(THE USER) GETS THE MAXIMUM MEMORY(RAM) FOR THE BUCK HE PAID FOR !!!

No ... techie ... but is it even possible in theory to have a memory footprint as low as my full blown latest linus desktop OS ?

At least could there be any hope on this ... @feherneoh ... go ahead burst my bubble !!!

feherneoh commented 4 years ago

I really hope @topjohnwu ... and his Magisk team and your efidroid team could hit a breakthrough implementation of a single multiboot solution ... something like the Grub we use on Linux machines ... I like the fact that Magisk a systemless implementation does not have to be adpated to each and every device ... just flash a single file across all devices ... a similar multiboot solution a single application(installer) that will apply to all android devices would be such a great option to have !!! I really Hope ... for i am just human !!!

That's just impossible

I know i fear ... probably it may be possible if google decides to soften and allow it such tech !!!

Still won't help, as for an usable dualboot that doesn't fry your device over time you need pre-kernel code, and there is just no way to make that completely universal. You MUST use builds customized for specific devices. Magisk can be universal, because anything not part of userspace it just pre-patches at install

:( :( :( ... oh ok ... my worst of fears ... i am no techie ... but seems you people know what you are talking about ... and i totally understand what you are saying conceptually !!!

Iphone 11 has 4 GB RAM ... and we have Android phones with 12 GB RAM ... What the Hell !!! JUST WISH SOMEBODY CAME UP WITH A BETTER WAY TO HANDLE MEMORY(RAM) ... AND I(THE USER) GETS THE MAXIMUM MEMORY(RAM) FOR THE BUCK HE PAID FOR !!! No ... techie ... but is it even possible in theory to have a memory footprint as low as my full blown latest linus desktop OS ?

At least could there be any hope on this ... @feherneoh ... go ahead burst my bubble !!!

I wouldn't go that far as to say one or another OS handles RAM better, they just do it differently. iOS is in a bit of a better situation, because of the native apps, so they are only caching the actual code, not both the dalvik bytecode and the output of JIT, so their memory footprint can be lower even with caching. Also AFAIK iOS apps aren't installed in a compressed state, so they don't have to decompress everything to RAM.

Comparing to desktop distros isn't the best idea, as generally one has faster disk IO on those devices. eMMC and UFS is just not as fast as an NVMe SSD. so caching as much as one can does drastically improve performance.

aiamuzz commented 4 years ago

I wouldn't go that far as to say one or another OS handles RAM better, they just do it differently. iOS is in a bit of a better situation, because of the native apps, so they are only caching the actual code, not both the dalvik bytecode and the output of JIT,

hmmmm ... ok ... i had a pretty rough guess of the issues with whole java and vm stuff under the android's hood ... you are confirming it ...

Also AFAIK iOS apps aren't installed in a compressed state, so they don't have to decompress everything to RAM.

just a thought now that Android devices are coming up with higher storage 64GB / 129 GB ... how much of a memory gain(lesser RAM) will android start to benefit if google decided to install apps in a decompressed state ... they could create bigger data partitions ?

Comparing to desktop distros isn't the best idea, as generally one has faster disk IO on those devices. eMMC and UFS is just not as fast as an NVMe SSD.

my laptop does not have a SSD ... plain old magnetic harddisks ...

@feherneoh ... thanks for sharing and bearing with my noob questions ... its always good to know the underlying concepts !!!

feherneoh commented 4 years ago

just a thought now that Android devices are coming up with higher storage 64GB / 129 GB ... how much of a memory gain(lesser RAM) will android start to benefit if google decided to install apps in a decompressed state ... they could create bigger data partitions ?

They install apps compressed because this way they have to check the signatures of ONE file, not every file the app contains my laptop does not have a SSD ... plain old magnetic harddisks ...

Might still be faster than some of the eMMC chips, also was WAY faster than the old MTD NAND chips used in Android devices before eMMC

aiamuzz commented 4 years ago

They install apps compressed because this way they have to check the signatures of ONE file, not every file the app contains

seems like my wallet is not getting lighter anytime soon (or may be never) as far as paying for RAM so google make merry !!!

@feherneoh ... are there any other mobile OS's which have a different design when it comes to handling RAM closer to iPhone's implementation ... SailfishOS, UbuntuTouch, and any other OS that's in alpha/beta stages ... which respects hardware and uses it judiciously ?

At least i can actively watch those OS'es and hedge on them to run on my hardware ... should they become viable and gain traction !

feherneoh commented 4 years ago

They install apps compressed because this way they have to check the signatures of ONE file, not every file the app contains

seems like my wallet is not getting lighter anytime soon (or may be never) as far as paying for RAM so google make merry !!!

@feherneoh ... are there any other mobile OS's which have a different design when it comes to handling RAM closer to iPhone's implementation ... SailfishOS, UbuntuTouch, and any other OS that's in alpha/beta stages ... which respects hardware and uses it judiciously ?

At least i can actively watch those OS'es and hedge on them to run on my hardware ... should they become viable and gain traction !

Actually I prefer Google's way of doing it. Not counting the cache the average memory usage without apps is around 1.5GB on Pie, so on a 6GB device one still can use 4.5GB just fine

aiamuzz commented 4 years ago

Actually I prefer Google's way of doing it. Not counting the cache the average memory usage without apps is around 1.5GB on Pie, so on a 6GB device one still can use 4.5GB just fine

oh ... no way not on my device running pie ... can you tell me how i can achieve 1.5 GB memory and release 4.5 GB ... I would be greatful ...

@feherneoh ... how much memory do android services that are responsible for notifications take up ?

another thing that's been going on in my head is ... the possibility of ... say when i quite a particular app ... all traces of that app be erased from the memory zero chaching ? at least that way my hardware resources will be well respected ...

in linux services barely use up any memory ... how does it compare with services(if there is such a concept) something that enables alerts and notifications be received in real time on android ?

PS : @feherneoh ... will you please share your telegram ID ... you are a bundle of knowledge ... a summary (crash course on concepts ) ... also this thread is digressed from the place and the subject it began with ... :)

apologies @M1cha ...

feherneoh commented 4 years ago

Actually I prefer Google's way of doing it. Not counting the cache the average memory usage without apps is around 1.5GB on Pie, so on a 6GB device one still can use 4.5GB just fine

oh ... no way not on my device running pie ... can you tell me how i can achieve 1.5 GB memory and release 4.5 GB ... I would be greatful ...

Again, that's NOT counting the caches, but those are freed anyways when an app requests memory. Just think about it like this: System assumes you might open a specific app after you receive a notification from it, so it loads it to cache. If you actually open it, then it can just grab it from memory instead of loading it that moment. If you don't open it, it will still keep it loaded, but throw it out when the memory is starts getting full. I cannot see why it would have to keep the overall memory usage low, when caches can be dropped any time more RAM is needed

@feherneoh ... how much memory do android services that are responsible for notifications take up ?

NORMALLY push notifications for most apps handled by a common Google service, so not that much. That's how you can receive message notifications even when the messaging app itself is actually killed, so you don't have to keep EVERY app open all the time.

another thing that's been going on in my head is ... the possibility of ... say when i quite a particular app ... all traces of that app be erased from the memory zero chaching ? at least that way my hardware resources will be well respected ...

Why would NOT using the RAM be a way of respecting hardware? If it's not used, it's wasted. That's why Android tends to cache as much as possible, so your extra RAM is always in use.

in linux services barely use up any memory ... how does it compare with services(if there is such a concept) something that enables alerts and notifications be received in real time on android ?

On Android "services" are parts of the systems. Anything else running in the background is NOT a service, but a full app. So like if an app doesn't use the built-in notfication handling service, it has to be FULLY loaded to be able to give you any notifications.

PS : @feherneoh ... will you please share your telegram ID ... you are a bundle of knowledge ... a summary (crash course on concepts ) ... also this thread is digressed from the place and the subject it began with ... :)

apologies @M1cha ...

My Telegram username is the same as here

GreenLunar commented 1 year ago

I think it's a good opportunity to look into othre operating systems such as NixOS, UBports and postmarketOS.

There's a growing interest in dual booting in this community.

I'm using postmarketOS for more than a year now and I don't want to go back to other tracking-driven and non-real-linux systems.

M1cha commented 1 year ago

Well if it's just about porting U-Boot that sounds like an interesting project with minimal effort. Efidroid was complex due to the following things that you don't need for that if I'm right:

GreenLunar commented 1 year ago

Well if it's just about porting U-Boot that sounds like an interesting project with minimal effort.

Then let's do it. I'm hording devices for test lab and more devices to port U-Boot into. I'm not sure how minimal it is because I'm not experienced with it, yet.

M1cha commented 1 year ago

Depends on the device. QCOM stopped supporting LK and their edk2 port is under NDA meaning we have no easy source of minimal drivers that can be used in a bootloader environment.

That means it's up to what U-Boot supports right now and about how capable we are in understanding the Linux drivers.

In theory, MMC and it's dependencies are the only thing we need though. Those dependencies can be a lot though depending on the architecture:

It's all reasonable but we definitely need a way to receive log messages so drivers can be developed and debugged properly.

For efidroid, a screen console was surprisingly effective since the first stage bootloader usually initializes the screen already to show a bootlogo and we can simply write pixels to ram.

UART is still preferred though of course.

z3ntu commented 1 year ago

Qualcomm has edk2 open source repository, e.g. https://git.codelinaro.org/clo/la/abl/tianocore/edk2/-/tree/LA.UM.9.12.r1-15100-SMxx50.QSSI13.0 but I don't think it includes any actual device drivers, they're in the proprietary xbl part (and not abl which this is the source for)

luka177 commented 1 year ago

@z3ntu Qcom bootloader is spit to 2 parts XBL and ABL, XBL stands for extensive bootloader, and its goal is to implement device drivers and do basic platform init. XBL is proprietary. While ABL stands for Android Boot Loader, its goal is to implement Fastboot nad to actually boot linux kernel with initrd and dtb, ah and yea it implements abootimg related stuff. You did link abl and yea it is opensource. Hope that helps :D

luka177 commented 1 year ago

About android dualboot yat there are tones of problems, everything M1cha did mention + some new things like GKI. Anuway i do try to implement new dualboot solution https://github.com/Android-Boot-Manager it is different from efidroid, currently we do support only android+non-android (UBports SailfishOS, whatever) but 0.4 version should change it. Main difference is, we do not bind this solution to any specific bootloader, ABM currently do support mtk-lk, qcom-lk (lk2nd) and now even u-boot. We are working on renegade uefi support too https://github.com/Android-Boot-Manager/droidboot_device_renegade-uefi Our non-platform specific part implements lvgl based gui, lwext4 rw fs driver, gpt parsing and config parsing. Platform part should implement blkread/write functions, fb flush function, function to check buttons state and linux booting from RAM. Currently ABM is at rather early but kinda of working stage.