search

guysoft / CustomPiOS

A Raspberry Pi and other ARM devices distribution builder
GNU General Public License v3.0
515 stars 149 forks source link

Issues with building Armbian images #7

Open TheAssassin opened 7 years ago

TheAssassin commented 7 years ago

I tried to build an Armbian image. As suggested by the variants directory, this is officially supported.

Issues I noticed:

Regarding the missing dependencies, someone has to invest some time into testing this, which I don't have at the moment. PRs welcome I guess.

(If the Ubuntu build shouldn't work either, I'll considering taking the time for further tests with Debian as well, but will try to fix at least the Ubuntu build.)

guysoft commented 7 years ago

Strange, it worked for me with the ExampleOS. See https://github.com/guysoft/CustomPiOS/pull/2 where @huelvayork claims it works for them too. It might be a specific build of Armbian. Also, you can append stuff to the Armbian variant: https://github.com/guysoft/CustomPiOS/tree/CustomPiOS/src/variants/armbian

TheAssassin commented 7 years ago

Well, I had to at least change the BASE_ZIP_IMG setting to something more generic to make it work at all. Regarding https://dl.armbian.com/, many images use the .7z format, and don't follow the naming convention used at the moment.

The fix implemented in #8 doesn't harm any of the implemented at least, so it's good it's merged. Maybe you could try to build an image for one of those Armbian images? You will probably see the issues I experienced as well.

guysoft commented 7 years ago

I can't really test those images without the devices. You can make it glob rhe conventions you need from the variant or with config.local

TheAssassin commented 7 years ago

Ok, I wasn't aware of the config.local option, but I'll think about improving the glob.

Meanwhile, you can test any built image from your computer as well. I've booted into mine from qemu without any issues, and reproduced the login issue. I'm not saying you should, but this is how it could be done.

TheAssassin commented 7 years ago

Next issue:

adduser: Only root may add a user or group to the system.

I am not quite sure yet why this occurs. This is the Ubuntu build. I'm going to fix it soon.

If you didn't guess already, I should mention I am trying to build OctoPi.

guysoft commented 7 years ago

How are you booting from qemu? See: https://github.com/guysoft/FullPageOS/issues/150 People there might wanna know how :)

TheAssassin commented 7 years ago

How are you booting from qemu? See: guysoft/FullPageOS#150 People there might wanna know how :)

It's not very stable, and it's quite some work (you need to extract and provide the right kernels to qemu etc.), I'll have to test that a bit more in order to be able to provide instructions.

In the meanwhile, I found out what issues I had with the adduser command: I tried to log in as pi, but that user is added by your scripts, which I didn't realize in the first place. When logging in with the Armbian default credentials (root and 1234), the login process works fine. For Armbian, it'd be best if the pi user wouldn't be able to login by default, and function more like a system/daemon user (maybe use a completely different username without a password set), otherwise, this is a security issue. This requires some more investigation in my opinion.

As soon as I get to figuring out how to use the damn nmcli, I'll provide information about my experience with the image I built.

guysoft commented 6 years ago

Hey, I manage to run builds now using both Ubuntu and Debian Armbian builds. So if no problems I am going to close this.

huelvayork commented 6 years ago

fine!

2017-12-26 9:20 GMT+01:00 Guy Sheffer notifications@github.com:

Hey, I manage to run builds now using both Ubuntu and Debian Armbian builds. So if no problems I am going to close this.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/guysoft/CustomPiOS/issues/7#issuecomment-353937231, or mute the thread https://github.com/notifications/unsubscribe-auth/AAqjPxzsqhwCfW_tyS8oQrn239DjAt-Nks5tEKxTgaJpZM4P5JbF .

-- Un saludo, David Prieto.

guysoft commented 6 years ago

Was waiting for @TheAssassin to reply who is OP. Closing if no reply

TheAssassin commented 6 years ago

Ah. I didn't think I would have to take any further action. I'll test it and report back.

guysoft commented 6 years ago

Well the issue is open, but it's also a free open source project. So I don't know if it's works unless test it. We are volunteers here :)

TheAssassin commented 6 years ago

There's information missing how to build Armbian images (i.e., what to call with which parameters). I think https://github.com/guysoft/CustomPiOS/wiki/Building needs some information on that.

What I did was:

Seems to work now, the image is extracted, I was quite happy about it. However, right after the fdisk calls, this happens:

++++ losetup -f --show -o 4194304 Armbian_5.35_Orangepizero_Ubuntu_xenial_next_4.13.16.img
losetup: Armbian_5.35_Orangepizero_Ubuntu_xenial_next_4.13.16.img: failed to set up loop device: Permission denied

Re-calling this script with sudo fixed the issues. But, as the script sudos itself in other places, that command should be prefixed with sudo as well, I'd say.

At least I got a file called /workspace-armbian/Armbian_5.35_Orangepizero_Ubuntu_xenial_next_4.13.16.img with a size of 1.8G, which looks quite valid to me. I couldn't test it yet (took my printer and its electronics apart, which the Orange Pi Zero is part of), but so far, so good.

As you might have seen, there's a few minor issues which are still left. But it's much better than when I first tried it, so thanks for improving this so far.

guysoft commented 6 years ago

We could add a page in the wiki with this info. Because ot seems there is a guide needed for armbian builds

TheAssassin commented 6 years ago

Yeah. The "actual" official build guide is in the wiki, too (although that should IMO be moved into the README, or be highlighted better in the README, I had some trouble finding it).

guysoft commented 6 years ago

We can link to it from the readme

TheAssassin commented 6 years ago

... or move this to the README. I guess this is what everyone is interested in, it's literally the most essential part of the documentation of this project. Half of this is in there anyway, so moving it to the README entirely would eliminate those duplicate docs, too. Also, in my experience, it's easier to keep a README up to date with the code than a wiki page, everything's available offline from the same directory.

guysoft commented 6 years ago

Ok, will also accept a pull request for the readme. The reason I moved it to the wiki was that I had duplication in OctoPi, FullPageOS and every other distro I made

TheAssassin commented 6 years ago

Well, you can still link to the README from those places, it doesn't really matter whether it's in the wiki or in the README. But well, I'll stop argumenting about semantics. I'll try to send a PR.

ChrisP-Git commented 6 years ago

Hi,

I think I will share my exeprience here. I'm trying to build a distribution based on Debian based armbian for an Odroid C1. I created a headless vm from scratch, based on Debian 8.2 64, for compilation. During netinst phase, I created a standard user named octopi. Here is my step by step:

After installation, logged as root:

apt-get install sudo 
sudo adduser octopi sudo

Then logged as octopi:

sudo apt-get install gawk util-linux realpath qemu-user-static git p7zip-full 

git clone https://github.com/guysoft/CustomPiOS.git 
git clone https://github.com/guysoft/OctoPi.git 

cd CustomPiOS/src/variants/armbian
sudo modprobe loop

cd OctoPi/src/
mkdir image-variants
cd image-variants
wget https://dl.armbian.com/odroidc1/archive/Armbian_5.35_Odroidc1_Debian_jessie_default_3.10.107.7z

cd ..
../../CustomPiOS/src/update-custompios-paths

python 3 is not available by default, I had to add it to avoid build error, with: apt-get install python3

Then the strange thing is I cannot launch the build without sudo, or I get errors of fsdisk not available. So I used: sudo ./build_dist armbian

img is fully created, the odroid c1 boot, but I can't get it by network (no DHCP lease requested). However while connected physically, I can login using pi/raspberry but just after login that a new user is requested but failed due to adduser unavailable, and process just loop here.

I could go to the Ubuntu flavor, but I prefer if possible to stay on headless Debian.

ChrisP-Git commented 6 years ago

Regarding the adduser issue, I think it's implemented by this script: https://github.com/armbian/build/blob/a36c0f137507e41caf7371d0d4d6cc49d464ebec/packages/bsp/common/etc/profile.d/check_first_login.sh

So probably it could be bypassed by removing the file /root/.not_logged_in_yet, does that make sense ? Where could this command be added ? I will test it but do not know how and where to implement it.

TheAssassin commented 6 years ago

@ChrisP-Git I remember a similar issue, which was solved by installing adduser, and I believe I had sent a PR back then...

I won't get to testing this with real hardware any soon, but I will let you know once I do. Maybe check the commit history? All that is needed is something to install the adduser command.

ChrisP-Git commented 6 years ago

@TheAssassin thanks for the head-up. I tried to find a PR concerning adduser but didn't find any (maybe I missed it). Anyway I'm not sure this is really needed, as it's just required by the first connection on the device, and the image allready have 2 users, root and pi. However I tried to remove the file /root/.not_logged_in_yet from the sd card and it successfully bypassed this.

The issue I have now is that eth0 is not started by default. tried something on a fork but it didn't fixed it.

TheAssassin commented 6 years ago

The idea is to get rid of the default username/password combination. There's 1000s of similar devices with SSH servers available on the internet (via port forwardings). The only way to get rid of those, though, is to force people to add another user with different values on first use. A lot of users would otherwise not change either value for convenience.

It is explicitly recommended to not "solve" this problem like you suggest, but rather fix the dependency issue.

ChrisP-Git commented 6 years ago

I get your point, and preventing end user to simply redirect ssh port is a way to go. But the first adduser triggered is not replacing the 'pi' user already present in the image, it's just adding another one if I'm not mistaken by the process. And by the way, by default SSH is enabled for root user on armbian image.

guysoft commented 6 years ago

I suggest we start with a basic guide how to do it at all and then expand it. Before changing the default security in Armbian.

ChrisP-Git commented 6 years ago

Well, it's a bit tricky as the standard initialisation phase on Armbian should be to logon as root, change the root password as requested, then create a new standard user.

However for OctoPi a prepopulated user (pi) is required to run the OctoPrint processes then created by the build process. And this is the process for the standard Raspbian image, and the others variants, right ?

So should this creation of user 'pi' inside the build be reverted and finding a way to use the user created during first run to execute the OctoPrint processes ?

I'm not saying that a way is better than the other, but I am a bit confused.

TheAssassin commented 6 years ago

@guysoft to users of this project, it won't matter whether the default Armbian behavior is changed or not, in my opinion. I would say they prefer it to be consistent across distros, actually.

arthurwolf commented 6 years ago

I've read this thread and the previous one, and i'm very confused what the status of this is. Let's take the most basic thing : I have a Orange Pi and I want an Octopi setup. What do I do ? I found an image but it's from 2016. What if I want a recent image ? Is there somewhere I can download it ? If there is no pre-made image ( you really should make them like for the normal octopi ), what do I need to do ? Build it myself is what I would guess, but where are the instructions to do that ? I can't find them easily. Sorry if this is located somewhere obvious, I didn't find.

indrora commented 5 years ago

Just tried to build for Armbian.

  1. the DNS server that is used no longer exists.
  2. The image generation fails in multiple places. for instance, Avahi can't be installed because the user for Avahi can't be added:
You are required to change your password immediately (root enforced)
chfn: PAM: Authentication token is no longer valid; new one required
adduser: `/usr/bin/chfn -f Avahi mDNS daemon avahi' returned error code 1. Exiting.
dpkg: error processing package avahi-daemon (--configure):
 subprocess installed post-installation script returned error exit status 1
Processing triggers for libc-bin (2.24-11+deb9u3) ...

and for some reason, pip fails: 

+ pip install pip --upgrade
Collecting pip
  Downloading https://files.pythonhosted.org/packages/46/dc/7fd5df840efb3e56c8b4f768793a237ec4ee59891959d6a215d63f727023/pip-19.0.1-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 9.0.1
    Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside environment /usr
Successfully installed pip-19.0.1
+ pip install virtualenv --upgrade
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main
+ exit 1
guysoft commented 5 years ago

@indrora Can you please provide the base images you are using to build?

indrora commented 5 years ago

It's Debian for Orange Pi Zero: https://dl.armbian.com/orangepizero/Debian_stretch_next.7z

The DNS server issue isn't tied to a specific image, so I'll go put in a PR for that.

On Mon, Jan 28, 2019, 01:56 Guy Sheffer <notifications@github.com wrote:

@indrora https://github.com/indrora Can you please provide the base images you are using to build?

On Jan 28, 2019 01:56, "Guy Sheffer" notifications@github.com wrote:

@indrora https://github.com/indrora Can you please provide the base images you are using to build?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/guysoft/CustomPiOS/issues/7#issuecomment-458044773, or mute the thread https://github.com/notifications/unsubscribe-auth/AAcuOLPajp8U6c7irZFA4ITXaVwkLXQnks5vHrtEgaJpZM4P5JbF .

indrora commented 5 years ago

Reference on Pip: This is a known thing and we should not be installing Pip like this?

see here: https://github.com/pypa/pip/issues/5599

indrora commented 5 years ago

Just noticed something: Pip gets installed, upgraded, virtualenv installed, then pip immediately removed. This leaves things in an inconsistent state.

This generally isn't safe and it's generally safer to use the distribution provided version of pip, virtualenv.

guysoft commented 5 years ago

@indrora I can change it to:

wget https://bootstrap.pypa.io/get-pip.py -O - | python3

Would that work?

guysoft commented 5 years ago

Looking at it, I think the pip install was there from either legacy code or something with the iptables, I suggest you just remove all of that, confirm it works without it, and then we commit that bit out.

luxflow commented 4 years ago

@guysoft which armbian distro version does CustomPIOS target for? I think it should be noted somewhere (currently jessie?, it seems armbian download page consider jessie legacy)

I have to remove NetworkManager to get wifi work becuase NetworkManager control network after armbian buster. (systemctl disable NetworkManager)

armbian has option for changing boot filesystem other than ext4 CustomPiOS may provide similar experience like raspberry pi to change settings (fat partition for /boot) what do you think about it? https://github.com/armbian/build/blob/1bf608e1ca21ef2c3c650689f3645832ec728da9/lib/configuration.sh#L88 https://docs.armbian.com/Developer-Guide_User-Configurations/#user-provided-configuration

guysoft commented 4 years ago

@luxflow So since this issue has been open. I've been playing around with the idea of making armbian stuff as a variable rather than a variant (and have the variant just set the variable).

In latest devel there is support for Ubuntu like this.

export BASE_DISTRO=ubuntu

There is a new variable BASE_DISTRO where the default is:

BASE_DISTRO=raspbian

Then we could have 

BASE_DISTRO=armbian_debian

and

BASE_DISTRO=armbian_ubuntu

Do you have an image you are building? I could test it and start implementing the new way to build, which should be easier to maintain.

luxflow commented 4 years ago

@guysoft here is what I have done to images, just try it, works well for me

https://github.com/luxflow/CustomPiOS/releases/tag/buster https://github.com/luxflow/OctoPi/releases/tag/0.17.0

guysoft commented 4 years ago

@luxflow Cool, we should really merge your changes in to mainline. Why are you disabling network manager? Any chance you will set up a PR?

luxflow commented 4 years ago

@guysoft IMO, network manager is useless for our case As I know, it is originally intended for laptop or desktop usecase. raspbian don't use NM by default armbian use NM because it is good choice for inexperienced users. https://forum.armbian.com/topic/6104-network-manager-woes/ armbian has NM based headless "onetime" wifi setup (armbian_first_run.txt) https://github.com/luxflow/CustomPiOS/commit/2979c8a4848a739c6cadbf4d404ca7d8ffb6d4ea#diff-d41311dfd21fdaa3fcceec5294fa29acR34 but I prefer wpa_supplicant.conf method which can change wifi setup multiple times

So why not disabling it?

I will PR soon

indrora commented 4 years ago

Fedora Server uses NetworkManager on ARM as well, as does CentOS. They've advised against networkd for a lack of maturity.

Raspbian uses others because of any number of reasons, including long standing compatibility things?

I would advise against removing it, given other distributions usage.

On Sun, Jan 26, 2020, 21:00 luxflow notifications@github.com wrote:

@guysoft https://github.com/guysoft IMO, network manager is useless for our case As I know, it is originally intended for laptop or desktop. raspbian don't use NM by default armbian use NM because it is good choice for inexperienced users. https://forum.armbian.com/topic/6104-network-manager-woes/ armbian has NM based headless "onetime" wifi setup (armbian_first_run.txt) luxflow@2979c8a#diff-d41311dfd21fdaa3fcceec5294fa29acR34 https://github.com/luxflow/CustomPiOS/commit/2979c8a4848a739c6cadbf4d404ca7d8ffb6d4ea#diff-d41311dfd21fdaa3fcceec5294fa29acR34 but I prefer wpa_supplicant.conf method which can change wifi setup multiple times

So why not disabling it?

I will PR soon

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/guysoft/CustomPiOS/issues/7?email_source=notifications&email_token=AADS4OHGCI3YQT73STZ35RLQ7ZS6PA5CNFSM4D7ES3C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ6J2ZY#issuecomment-578592103, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADS4OHZTRS5FX44MU3P5B3Q7ZS6PANCNFSM4D7ES3CQ .

luxflow commented 4 years ago

systemd-networkd seems off-topic thing isn't it? I prefer ifupdown than NetworkManager since it gives similar user experience like raspbian. And as I said earlier, armbian only support onetime headless setup. whenever change wifi ssid, password. this was quite nuisance for me I'm not strongly against NM, so using armbian default behavior is OK for me

and I found why armbian drop fat boot partition and use ext4 by default https://salsa.debian.org/cklein-guest/dpkg/merge_requests/1 so whenever kernel update, dpkg hard link old file for backup, but fat doesn't have hard link. raspbian solve this problem using dpkg-divert hack. I didn't noticed this since I didn't upgrade kernel for orange pi should have to freeze armbian packages, preventing from user accidentally break system

personally, I think we should have to deal with all of these problems from a practical point of view 1) do we need to prefer NetworkManager at the expense of similar user experience of raspbian? 2) do kernel upgrade matter than fat bootfs easy setup for 99% user?

I'm not sure which way is better

guysoft commented 4 years ago

I rather stuff stay similar to the Raspberry Pi initial configuration (well with the CustomPiOS edition that is part of the network module, such as having a wifi text file with presets).. So if they don't have network manager by default its legit. It could be added as an optional module in CustomPiOS if others need it. So the Armbian base setup should remove it.

So I vote:

  1. no
  2. I am not suer I understand that question - I think kernel upgrades should not be default. We could have a upgrade-kernel module that would take care of that.
guysoft commented 4 years ago

68 just went it, it might solve some of your issues.