search

openDsh / dash

Join us on Slack! https://join.slack.com/t/opendsh/shared_invite/zt-la398uly-a6eMH5ttEQhbtE6asVKx4Q
GNU General Public License v3.0
238 stars 69 forks source link

Install overhaul #165

Open breakingspell opened 1 month ago

breakingspell commented 1 month ago

Various improvements and restructuring of the install process. Targeting Ubuntu 22.04 and Debian 12 (Raspberry Pi OS 2024-03).

DONE:

Bonus idea:

Related issue (if applicable): fixes # https://github.com/openDsh/dash/issues/63 https://github.com/openDsh/dash/issues/127 https://github.com/openDsh/dash/issues/162

rsjudka commented 1 month ago

i like the idea :+1: cant wait to see this merged in!

breakingspell commented 3 weeks ago

Getting close to done with the overhaul: compiles and executes on both my Pi 3 and 4 with no hitches outside of the resource pinch on the 3. AA connection to the phone works over USB and Wireless (haven't tested wireless on the 3).

The RPi autostart methods all work, system daemon and desktop icon assuming the default DE package raspberrypi-ui-mods is installed and set to autologin. I personally prefer the more minimal xinit method that only requires the Xorg base, seems to be the quickest startup too.

I haven't tested the memory split or krnbt flags, they seem to be for even older boards than the 3. The GL module is a bit tricky, as it defaults to loading, and now the distinction seems to be between whether it uses real KMS or "fake" KMS, which also affects the display/brightness system.

From here I need to test ofono/noise cancellation, then clean up and prettify parts that lack "echo" outputs and comments.

parad0xnasha commented 2 weeks ago

This has hit a brick wall at 46% - (Pi 4 1GB) Went unresponsive for about 20min, and ticked over to 47% at which 762MB/907MB RAM utilization at this point, and has entered another period of unresponsiveness (for at least the last 7min). The main branch would only make it to ~650MB utilization before freezing. Was your testing performed on physical devices, or dev-environment-emulators? (dev-emulation seems to be a rather exhaustive workaround to compiling directly on the Pi). I'm quite surprised that this issue took so long to be reported as an actual problem that needed addressing. Willing to assist in any manner possible in relation to this issue, if that would be of assistance.

robert5974 commented 2 weeks ago

It has been tested on both physical and virtual. Few questions for you, what OS are you using? What is the error you received if any when it freezes? If there is no error, what is the last few lines of the install?

The installer was created and tested against Raspberry Pi OS 32-bit before the 64-bit OS was released and on most models of the Rpi 3 and 4. I don't recall anyone using a pi4 1gb so perhaps that is an issue as there was a portion that was different per model which we tried to capture every model

in use.

It isn't. So much of a surprise that the install failed as it does all the time depending on the board model, OS, package changes in the repos of the different os versions. There are many factors and we've dealt with them as they come up.

Robert Crowley

On Mon, Jun 10, 2024, 12:27 PM parad0xnasha @.***> wrote:

This has hit a brick wall at 46% - (Pi 4 1GB) Went unresponsive for about 20min, and ticked over to 47% at which 762MB/907MB RAM utilization at this point, and has entered another period of unresponsiveness (for at least the last 7min). The main branch would only make it to ~650MB utilization before freezing. Was your testing performed on physical devices, or dev-environment-emulators? (dev-emulation seems to be a rather exhaustive workaround to compiling directly on the Pi). I'm quite surprised that this issue took so long to be reported as an actual problem that needed addressing. Willing to assist in any manner possible in relation to this issue, if that would be of assistance.

— Reply to this email directly, view it on GitHub https://github.com/openDsh/dash/pull/165#issuecomment-2158819423, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLUDVFX7JREOM3U2DZUPQTZGXHXHAVCNFSM6AAAAABHNJRFR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJYHAYTSNBSGM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

breakingspell commented 2 weeks ago

Hello all, thanks for testing! To note, I have tested exclusively on Pi4 2GB and Pi3 1GB hardware with the same image (apart from an early VM test with 2GB RAM). I have a feeling the reason it's failing earlier on this branch is due to me raising the parallel make jobs (-j4). We may still need to lower this to -j2 for some packages.

I did find that raising swap space on the 1GB model to 1000M allowed it to finish the build, at the cost of long build time (over hour and a half on the Pi3 versus 25-30mins on Pi4 2GB). Without the extra memory, build would fail on the AASDK, qt-gstreamer, and Openauto builds, where qt-gstreamer seems to be the most resource intensive. A few more details here: https://github.com/openDsh/dash/issues/162#issuecomment-2146881516.

Also, wasn't aware they manufactured 1GB Pi4 models! The resource advisory at the start of the script filters out Pi4 and up, I need to remove that distinction so there's visibility on the memory requirements.

parad0xnasha commented 2 weeks ago

It has been tested on both physical and virtual. Few questions for you, what OS are you using? What is the error you received if any when it freezes? If there is no error, what is the last few lines of the install? The installer was created and tested against Raspberry Pi OS 32-bit before the 64-bit OS was released and on most models of the Rpi 3 and 4. I don't recall anyone using a pi4 1gb so perhaps that is an issue as there was a portion that was different per model which we tried to capture every model # in use. It isn't. So much of a surprise that the install failed as it does all the time depending on the board model, OS, package changes in the repos of the different os versions. There are many factors and we've dealt with them as they come up. Robert Crowley

Much appreciated for the extremely prompt reply. Since posting this, i went looking at various other implementations, and it seems that a Pi 4 2GB is the bare minimum as the "accepted norm". This would explain the behaviour i'm seeing - I think it's swapping to SD REALLY REALLY slowly. I have chugged along to 52% now... I'd suggest a hardware requirement be set with a higher RAM level, but you initially approached this problem getting 3B+'s which also have 1GB across the line.

Knowing what i didn't know 24hrs ago, i'd purchase a Pi 4 4GB to approach this task. I'm going to purchase one, and happy to report back, but i suspect my SD-card is getting hammered to an inevitable demise! This is running latest Raspbian OS 64bit (Full desktop), and i did try the main branch with Raspbian Lite CLI also, which got me a little further than the GUI version. I feel like if i let this run for another 8hrs or so, I wouldn't be surprised if it does eventually complete. Just a matter of whether everything else stays running alongside it (it's continued to suffer repeated periods of non-responsiveness) - measured by the time on the clock - last update was 17min prior to current.

robert5974 commented 2 weeks ago

For me, I always had trouble with adding the -j2 or -j4 so I always removed it and just used the make command. It worked every time without specifying that. It obviously took a longer time but I figured it was just my board or something like that. Perhaps it's worth a try.

Robert Crowley

On Mon, Jun 10, 2024, 1:02 PM Blazer the Lazer @.***> wrote:

Hello all, thanks for testing! To note, I have tested exclusively on Pi4 2GB and Pi3 1GB hardware with the same image (apart from an early VM test with 2GB RAM). I have a feeling the reason it's failing earlier on this branch is due to me raising the parallel make jobs (-j4). We may still need to lower this to -j2 for some packages.

I did find that raising swap space on the 1GB model to 1000M allowed it to finish the build, at the cost of long build time (over hour and a half on the Pi3 versus 25-30mins on Pi4 2GB). Without the extra memory, build would fail on the AASDK, qt-gstreamer, and Openauto builds, where qt-gstreamer seems to be the most resource intensive. A few more details here: #162 (comment) https://github.com/openDsh/dash/issues/162#issuecomment-2146881516.

Also, wasn't aware they manufactured 1GB Pi4 models! The resource advisory at the start of the script filters out Pi4 and up, I need to remove that distinction so there's visibility on the memory requirements.

— Reply to this email directly, view it on GitHub https://github.com/openDsh/dash/pull/165#issuecomment-2158880252, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLUDVDRCRCUV7E6PA4E6TLZGXL3TAVCNFSM6AAAAABHNJRFR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJYHA4DAMRVGI . You are receiving this because you commented.Message ID: @.***>

parad0xnasha commented 2 weeks ago

Hello all, thanks for testing! To note, I have tested exclusively on Pi4 2GB and Pi3 1GB hardware with the same image (apart from an early VM test with 2GB RAM). I have a feeling the reason it's failing earlier on this branch is due to me raising the parallel make jobs (-j4). We may still need to lower this to -j2 for some packages.

I did find that raising swap space on the 1GB model to 1000M allowed it to finish the build, at the cost of long build time (over hour and a half on the Pi3 versus 25-30mins on Pi4 2GB). Without the extra memory, build would fail on the AASDK, qt-gstreamer, and Openauto builds, where qt-gstreamer seems to be the most resource intensive. A few more details here: #162 (comment).

Also, wasn't aware they manufactured 1GB Pi4 models! The resource advisory at the start of the script filters out Pi4 and up, I need to remove that distinction so there's visibility on the memory requirements. Well this would certainly explain the oversight! I'm glad i was able to provide something towards a resolution at least anyway!

For me, I always had trouble with adding the -j2 or -j4 so I always removed it and just used the make command. It worked every time without specifying that. It obviously took a longer time but I figured it was just my board or something like that. Perhaps it's worth a try. Robert Crowley

For now i'll stick to the advice offered by the man himself, dropping the concurrent jobs AND increasing the swap space in order to at least get a feel for this dashboard solution, but i think the inevitable outcome would be using a 4GB Pi. I think you've checked off the box of PI integration issues now, and without seeing 1GB operating beside a 2GB/4GB, I can't say for certain, but it does seem that a minimum requirement suggestion be placed somewhere more prominent (not only as remnants of an issue that once upon a time on Github), but if it is indeed the case then it needs to be said loud enough to drown mention of lesser HW unless knowingly stepping into that arena, Personally i dont believe this to be a uniquely OpenDash problem. Only the "thou shall not be named paid version" is the only software that sets hardware minimums, because they have to provide a level of service. Whereas OSS generally you want to reach as many users as possible... However, if i'd have known, given some indication prior to build/execution that ideally you want model X for smooth operation, i'd not have wasted the time and money. Project goes back onto pause now for me, until finances are such that i can freely purchase a 4gb, with little ROI - So i very well may consider abandoning the project, it's becoming much more than what i want/need, so i cant justify more costs. But yes, be nice if some people mentioned something about how troublesome 1GB devices can be, not just for this solution, but across the board

breakingspell commented 2 weeks ago

I've just learned that make without flags will default to using all available processing units it can (value returned with nproc, on both RPI 3 and 4 this is 4 threads), so the changes I made there are pretty useless. AASDK had -j2 set to begin with, but the rest can be default, I'll set them all back.

Only the "thou shall not be named paid version" is the only software that sets hardware minimums, because they have to provide a level of service.

:joy: As a former user of said software, no comment. The only real thing I notice we're lacking now to be feature-equal is Equalizer (been testing pulseeffects in my car, it does "work" but it's all over the place, no gui controls etc)

parad0xnasha commented 2 weeks ago

I've just learned that make without flags will default to using all available processing units it can (value returned with nproc, on both RPI 3 and 4 this is 4 threads), so the changes I made there are pretty useless. AASDK had -j2 set to begin with, but the rest can be default, I'll set them all back.

I assume this isn't very far from being merged into the openDsh/dash tree? Given the info learned, and potential hazards sufficiently handled

parad0xnasha commented 2 weeks ago

😂 As a former user of said software, no comment.

It was so moving, it put you onto a course of developing your own free alternative... Wow, i must really be in for a treat hahah!

Foolishly i purchased, so i'll give it a run, but it's unlikely to be the deployment choice. I should've dug deeper before making that terrible decision.

breakingspell commented 2 weeks ago

I assume this isn't very far from being merged into the openDsh/dash tree? Given the info learned, and potential hazards sufficiently handled

Just pushed the branch here, it now excludes the -j flags except on AASDK as before. Will test it on my Pi3 real quick with the raised swap.

I tested Ofono this weekend (was under the impression it was needed for Bluetooth HSF), and failing step is only for the noise cancellation module. Still need to write in the fix to insert the block into pulse config, and then write it into the uninstaller as well.

rsjudka commented 2 weeks ago

have been following along, just wanted to say appreciate all the work youve been putting in :raised_hands:

like the direction this is going in

breakingspell commented 2 weeks ago

I'm glad I could help! Installer just needed a spruce-up, the core application works great. Making some last minute line and comment cleanups tonight as I test in a fast VM, I'll mark the PR as ready once it's done.

It'll be tested on: -Rpi4 2GB -Ppi3 1GB+1GB swap -Github Actions (Ubuntu 22.04) -Base Ubuntu 22.04 Desktop VM (GNOME)

robert5974 commented 2 weeks ago

I'll probably be testing on my Pi4 4gb.

Robert Crowley

On Tue, Jun 11, 2024, 10:13 PM Blazer the Lazer @.***> wrote:

I'm glad I could help! Installer just needed a spruce-up, the core application works great. Making some last minute line and comment cleanups tonight as I test in a fast VM, I'll mark the PR as ready once it's done.

It'll be tested on: -Rpi4 2GB -Ppi3 1GB+1GB swap -Github Actions (Ubuntu 22.04) -Base Ubuntu 22.04 Desktop VM (GNOME)

— Reply to this email directly, view it on GitHub https://github.com/openDsh/dash/pull/165#issuecomment-2161971148, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLUDVFYE7XXU6BEOBXZXK3ZG6VDLAVCNFSM6AAAAABHNJRFR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRRHE3TCMJUHA . You are receiving this because you commented.Message ID: @.***>

breakingspell commented 2 weeks ago

Decided to move many of the "Rpi" options into an autostart.sh, they also work fine on the default Ubuntu desktop. rpi.sh only contains options specific to these units now. The uninstaller should be safer and I eliminated usage of cat where I could per linting advice. Didn't change more than I had to, I left most existing comments and spacing alone.

Confirmed thee three autostart options work well in Ubuntu, and just xinit for the Pi image (no need to install full DE). USB udev rules work for Ubuntu as well, wired Android Auto launches without issue (wireless should work with configuration).

Huge push, but this should be the final one for now unless we come up with oddities in testing. :+1:

robert5974 commented 2 weeks ago

Great work! I put my Pi back together on a fresh new dash fascia and will be testing it out today or tomorrow.

breakingspell commented 2 weeks ago

Uh oh, so i'm not sure if it's my BT adapter, Bluez, Ofono, or Pulse: turns out enabling Noise Cancellation causes the system to drop Handsfree calls 3 out of 4 times.

With just module-bluetooth-discover headset=ofono in place, Handsfree works each time (and is required), it's module-echo-cancel that seems to have trouble. The times it does engage a call, there's no noticeable difference in call quality (recordings to myself on google voice with the tiny car mic). I had only tested it with two calls earlier this week on the road and got lucky.

Before the setup overhaul, the noise cancellation step would fail due to shell permissions, so my system lacked that module all this time. My plan is to test picking and installing an older ofono version against the older pulseaudio (v12) to see if I can get both to work with no issues. If it's not clear, then I may consider commenting out the noise cancellation step until it's fixed.

robert5974 commented 2 weeks ago

I was able to clone the repo last night and let it install with no changes at all. Great work! I tested withPi4 2gb. I have a 4gb model as well and can probably test that as well but I'm not sure when I'll get the time until next week.

Robert Crowley

On Thu, Jun 13, 2024, 11:54 PM Blazer the Lazer @.***> wrote:

Uh oh, so i'm not sure if it's my BT adapter, Ofono, or Pulse: turns out enabling Noise Cancellation causes the system to drop Handsfree calls 3 out of 4 times.

With just the Ofono system in place, Handsfree works each time (and is required), it's the noise cancellation module that seems to have trouble. The times it does engage, there's no noticeable difference in call quality (recordings to myself on google voice with the tiny car mic). I had only tested it with two calls earlier this week on the road and got lucky. Before the setup overhaul, the noise cancellation step would fail due to shell permissions, so my system lacked that buggy module all this time.

My plan is to test picking and installing an older ofono version against the older pulseaudio (v12) to see if I can get both to work with no issues. If it's not clear, then I may consider commenting out the noise cancellation step until it's fixed.

— Reply to this email directly, view it on GitHub https://github.com/openDsh/dash/pull/165#issuecomment-2167161332, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLUDVAKBQFPVAREN4DBDJ3ZHJSN3AVCNFSM6AAAAABHNJRFR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRXGE3DCMZTGI . You are receiving this because you commented.Message ID: @.***>

breakingspell commented 2 weeks ago

One more small fix that I found today after driving on a fresh install: the xinit dependencies lacked x11-xserver-utils so xset s noblank wouldn't work without it. Quick push for that (squashed my commits way too early here :slightly_smiling_face: )

The Ofono component is much more confusing. I was under the impression that it was required for specific bluetooth handsfree modes on the Pi, but it looks like Pulse+Bluez are handling this: backend-native utilizes a2dp_source during media playback and headset_audio_gateway when on calls (the standard "PC" profile). This is noted in a relatively recent pull request: https://github.com/openDsh/openauto/pull/29.

When Ofono is enabled in pulse config, I get output in dmesg like Bluetooth: hci0: corrupted SCO packet. Found that Ofono itself is a GSM telephony component that seems it was to be used with a concept phone frontend app for Dash. Can't find any other mentions of it on this repository (and I don't have a personal Slack account to check discussion there), so i'll set the install flag to false unless there's a reason to continue troubleshooting it.

I can completely remove Ofono from the system and still make/receive calls inside wireless AA, so maybe HSP isn't necessary. Anyone else equipped to test handsfree calling with Wireless AA?

rsjudka commented 2 weeks ago

can't speak to ofono myself... if @icecube45 is around he might be able to say more about it 🙂