search

sudomesh / sudowrt-firmware

Scripts to build the sudo mesh OpenWRT firmware.
Other
73 stars 19 forks source link

build a build machine #111

Closed paidforby closed 6 years ago

paidforby commented 6 years ago

Recent difficulties and inconsistencies with our build process make me think it would be worth pursuing a streamlined, well-documented firmware build process. I'm interested to see what everyone thinks of this idea. List of (potential) requirements:

muxlux commented 6 years ago

how often do we compile & how long does it take? wondering if pi/bone is fast enough

On Mon, Oct 23, 2017 at 10:30 AM, grant_____ notifications@github.com wrote:

Recent difficulties and inconsistencies with our build process make me think it would be worth pursuing a streamlined, well-documented firmware build process. I'm interested to see what everyone thinks of this idea. List of (potential) requirements:

  • consistent, easy-to-find hardware (raspberry pi? beagle bone?)
  • light-weight, minified *nix OS and interface.
  • open-source, infinitely-reproducible configuration
  • large, local, reliable, and resilient storage space (RAID setup?)
  • minimal need for internet access (local copies of necessary open wrt packages? IPFS integration?)
  • use as mythical "flashing station" also? Please add other requirements and suggestions in the comments.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sudomesh/sudowrt-firmware/issues/111, or mute the thread https://github.com/notifications/unsubscribe-auth/ACLJ4saVp_dEBqFU67C3Wz0ERWdnpCg7ks5svM1CgaJpZM4QDNcZ .

paidforby commented 6 years ago

@muxlux Not too often. Take between three and five hours, depending if you want to build both home node and extender node firmware (current method you have to build both to build for extender node). I don't think processing power should be an issue, i would be more concerned with network speed and storage space (I borked my Ubuntu partition by rebuilding this firmware a few too many times). There is always the Banana Pi or the Odroid if we need a little more juice.

muxlux commented 6 years ago

banana pi m1 has sata

On Mon, Oct 23, 2017 at 10:56 AM, grant_____ notifications@github.com wrote:

Not too often. Take between three and five hours, depending if you want to build both home node and extender node firmware (current method you have to build both to build for extender node). I don't think processing power should be an issue, i would be more concerned with network speed and storage space (I borked my Ubuntu partition by rebuilding this firmware a few too many times). There is always the Banana Pi or the Odroid if we need a little more juice.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/sudomesh/sudowrt-firmware/issues/111#issuecomment-338743673, or mute the thread https://github.com/notifications/unsubscribe-auth/ACLJ4tKRFBPlUWhtinhO8LjCGGIGYCuHks5svNNFgaJpZM4QDNcZ .

Juul commented 6 years ago

Making this all build on a non-x86 platform would likely be even more of a headache and it would be slow. Why not use virtualization? I'm no fan of docker but what about something simpler like vagrant? That's what I use for building other non-x86 stuff and it works great.

paidforby commented 6 years ago

I'm all for making it virtual, but you'll always have a physical machine somewhere (unless we were to go the VPS route, but that raises different concerns). I've never used vagrant, but from my experiences with docker, virtual box, and hyper-v, i know that host system uptime, configuration, and resources are crucial. Sure we aren't pumping out firmware builds, but at the same time I don't want to tie up my computer's resources building firmware. A communal builds machine would open up the build process for more experimentation. We wouldn't want to make this the single source of all sudowrt firmware builds, so making it work on any machine is desirable, but this introduces possible host system variables. @Juul I see your concern with ARM-based hardware, I'd be interested to explore possible x86 systems if there are any as common and accessible as equivalent ARM systems.

andrewdollard commented 6 years ago

I've tried in the past to get the builds running on a cloud CI service, which I think would be the ideal outcome - just push a commit, wait a little bit, and download a fresh image, with no hardware to deal with. Plus most of the CI services have a free tier for open-source projects, and they generally run docker, which we're using.

The problem that I encountered is that the build takes so long that all of the services I tried timed out before it finished. At the moment I'm trying to do some intelligent docker caching to speed things up, but the feedback cycle is pretty long...

On Mon, Oct 23, 2017 at 7:04 PM, grant_____ notifications@github.com wrote:

I'm all for making it virtual, but you'll always have a physical machine somewhere (unless we were to go the VPS route, but that raises different concerns). I've never used vagrant, but from my experiences with docker, virtual box, and hyper-v, i know that host system uptime, configuration, and resources are crucial. Sure we aren't pumping out firmware builds, but at the same time I don't want to tie up my computer's resources building firmware. A communal builds machine would open up the build process for more experimentation. We wouldn't want to make this the single source of all sudowrt firmware builds, so making it work on any machine is desirable, but this introduces possible host system variables. @Juul https://github.com/juul I see your concern with ARM-based hardware, I'd be interested to explore possible x86 systems if there are any as common and accessible as equivalent ARM systems.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sudomesh/sudowrt-firmware/issues/111#issuecomment-338849455, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDy8Fqkvs-_0l4yNrgQMLVea5KWLTmCks5svUWugaJpZM4QDNcZ .

muxlux commented 6 years ago

3d printer's puter could be used but it wouldn't be accessible from outside of our firewall unless you're on the mesh (i think)

On Mon, Oct 23, 2017 at 7:42 PM, Andrew Dollard notifications@github.com wrote:

I've tried in the past to get the builds running on a cloud CI service, which I think would be the ideal outcome - just push a commit, wait a little bit, and download a fresh image, with no hardware to deal with. Plus most of the CI services have a free tier for open-source projects, and they generally run docker, which we're using.

The problem that I encountered is that the build takes so long that all of the services I tried timed out before it finished. At the moment I'm trying to do some intelligent docker caching to speed things up, but the feedback cycle is pretty long...

On Mon, Oct 23, 2017 at 7:04 PM, grant_____ notifications@github.com wrote:

I'm all for making it virtual, but you'll always have a physical machine somewhere (unless we were to go the VPS route, but that raises different concerns). I've never used vagrant, but from my experiences with docker, virtual box, and hyper-v, i know that host system uptime, configuration, and resources are crucial. Sure we aren't pumping out firmware builds, but at the same time I don't want to tie up my computer's resources building firmware. A communal builds machine would open up the build process for more experimentation. We wouldn't want to make this the single source of all sudowrt firmware builds, so making it work on any machine is desirable, but this introduces possible host system variables. @Juul https://github.com/juul I see your concern with ARM-based hardware, I'd be interested to explore possible x86 systems if there are any as common and accessible as equivalent ARM systems.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/sudomesh/sudowrt-firmware/issues/111# issuecomment-338849455, or mute the thread https://github.com/notifications/unsubscribe-auth/ABDy8Fqkvs-_ 0l4yNrgQMLVea5KWLTmCks5svUWugaJpZM4QDNcZ .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sudomesh/sudowrt-firmware/issues/111#issuecomment-338855395, or mute the thread https://github.com/notifications/unsubscribe-auth/ACLJ4v3Fg-M1wm410buYEqTp4YCro2rfks5svU6igaJpZM4QDNcZ .

jhpoelen commented 6 years ago

having a build server for the firmware sounds like a good idea. . . how far did we get on this? want to update my firmware on my home node, and trying to figure out whether to build my own firmware or pick an existing one.

muxlux commented 6 years ago

fwiw, the box connected to the 3d printer is always on & haz 2 nics. one is configured to the mesh & the other is 192.168.10.x (default nanobridge ip range)

On Sun, Nov 5, 2017 at 11:39 AM, Jorrit Poelen notifications@github.com wrote:

having a build server for the firmware sounds like a good idea. . . how far did we get on this? want to update my firmware on my home node, and trying to figure out whether to build my own firmware or pick an existing one.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sudomesh/sudowrt-firmware/issues/111#issuecomment-341999043, or mute the thread https://github.com/notifications/unsubscribe-auth/ACLJ4sZEQ3YlNwVamji5BQK0s4qj1O8zks5szg7ygaJpZM4QDNcZ .

Juul commented 6 years ago

I can put it on a computer that's always on. This is not an issue. Getting it actually working in a VM in a way that it can be set up on any (at least x86) computer without a bunch of hassle is the problem. I like vagrant over other systems for making it easy to build stuff because it is very very simple. It's literally just a vagrant script that configures the VM and a single bash script that does the one-time configuration. Take a look at this repo (never mind the huge readme).

jhpoelen commented 6 years ago

I've created a first pass at having travis push part of the firmware build machine image to docker hub, see .travis.yml. Also, see https://hub.docker.com/r/sudowrt/firmware .

so, now you can do docker pull sudowrt/firmware to pull latest firmware build machine image. Still not perfect, but able to re-use bandwidth provided by travis.

jhpoelen commented 6 years ago

Did some more refactoring of the build scripts to separate grabbing of external resources (repos/feeds etc) from the actual building. Travis is happy, still attempting to do a full build. Please review build_lib / build_only / build / build_pre script if anyone gets around to it.

jhpoelen commented 6 years ago

Was able to create home node binaries using 13814cd229c2ea11bbfb62300936b958e0c3ccee in combination with travis / docker . Extender node failed because I ran out of disk space.

to reproduce:

  1. clone https://github.com/sudomesh/sudowrt-firmware
  2. docker pull sudowrt/firmware (pull build machine image created by travis from dockerhub)
  3. docker run -v $PWD/firmware_images:/firmware_images sudowrt/firmware (kick off make/build)
  4. inspect [sudowrt-firmware dir]/firmware_images

Here's the time stats just for building ar71xx home node -

real 107m1.262s user 65m55.060s sys 9m12.568s

Juul commented 6 years ago

Woo! Hurray!

On Tue, Nov 21, 2017 at 8:34 AM, Jorrit Poelen notifications@github.com wrote:

Was able to create home node binaries using 13814cd https://github.com/sudomesh/sudowrt-firmware/commit/13814cd229c2ea11bbfb62300936b958e0c3ccee in combination with travis / docker . Extender node failed because I ran out of disk space.

to reproduce:

  1. clone https://github.com/sudomesh/sudowrt-firmware
  2. docker pull sudowrt/firmware (pull build machine image created by travis from dockerhub)
  3. docker run -v $PWD/firmware_images:/firmware_images sudowrt/firmware (kick off make/build)
  4. inspect [sudowrt-firmware dir]/firmware_images

Here's the time stats -

real 107m1.262s user 65m55.060s sys 9m12.568s

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sudomesh/sudowrt-firmware/issues/111#issuecomment-346083909, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHfgHRWrHmfefDg1SyVF6F13ggbM1Iaks5s4vuRgaJpZM4QDNcZ .

paidforby commented 6 years ago

Work'd for me also. The firmware is located in [sudowrt-firmware dir]/firmware_images/built_firmware/builder.ar71xx/bin/ar71xx and has n600 bin.

stats for home node:

real    124m35.957s
user    66m10.532s
sys 9m29.432s

stats for extender node:

real    37m30.702s
user    26m3.356s
sys 5m6.140s

Nicely done @jhpoelen

jhpoelen commented 6 years ago

Was just able to flash a MyNet N600 using firmware built using the travis/docker route and able to login through ssh (see below).

Now, hoping to resolve https://github.com/sudomesh/makenode/issues/16 to confirm makenode works as usual.

$ ssh root@172.22.0.1
BusyBox v1.23.2 (2017-11-21 21:45:39 UTC) built-in shell (ash)

  ._______.___    ._______.______  ._____  .___    .___ .______  ._____  
  :_ ____/|   |   : .____/:_ _   \ :_ ___\ |   |   : __|:      \ :_ ___\ 
  |   _/  |   |   | : _/\ |   |   ||   |___|   |   | : ||       ||   |___
  |   |   |   |/\ |   /  \| . |   ||   /  ||   |/\ |   ||   |   ||   /  |
  |_. |   |   /  \|_.: __/|. ____/ |. __  ||   /  \|   ||___|   ||. __  |
    :/    |______/   :/    :/       :/ |. ||______/|___|    |___| :/ |. |
    :                      :        :   :/                        :   :/ 
                                        :                             : 
 -------------------------------------------------------------------------
 sudo mesh v0.2 (fledgling)
 UNCONFIGURED - see https://github.com/sudomesh/makenode
                for configurator. 

                              based on OpenWRT 15.05 (Chaos Calmer)
 -------------------------------------------------------------------------
 “When your rage is choking you, it is best to say nothing.” 
                                            - Octavia E. Butler, Fledgling
 -------------------------------------------------------------------------
muxlux commented 6 years ago

grr, docker no worky w/lowlatency kernel :(

On Thu, Nov 23, 2017 at 11:13 AM, Jorrit Poelen notifications@github.com wrote:

Was just able to flash a MyNet N600 using firmware built using the travis/docker route and able to login through ssh.

Now, hoping to resolve sudomesh/makenode#16 https://github.com/sudomesh/makenode/issues/16 to confirm makenode works as usual.

$ ssh root@172.22.0.1 BusyBox v1.23.2 (2017-11-21 21:45:39 UTC) built-in shell (ash)

.____. ..____ . . . .__ . : / : ._/: \ : _\ : __ : \ :_ ___\ _/ : _/\ ___ : ___ /\ / \ . / /\ / _. / \ _.: __/ . ____/ . __ / \ ___ . __ :/ __/ :/ :/ :/ . __/ ___ ___ :/ . : : : :/ : :/

sudo mesh v0.2 (fledgling) UNCONFIGURED - see https://github.com/sudomesh/makenode for configurator.

                          based on OpenWRT 15.05 (Chaos Calmer)

“When your rage is choking you, it is best to say nothing.”

  • Octavia E. Butler, Fledgling

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sudomesh/sudowrt-firmware/issues/111#issuecomment-346685035, or mute the thread https://github.com/notifications/unsubscribe-auth/ACLJ4uae2r6kxwhYZt4ZiVx847aju3aQks5s5cPpgaJpZM4QDNcZ .

jhpoelen commented 6 years ago

Last night, I was able to get an internet connection through ad-hoc mesh connections at sudoroom through n600 and n750 running a recently built firmware, resolving https://github.com/sudomesh/sudowrt-firmware/issues/105.

Reason for delay was my lack of understanding on how to troubleshoot a home node in addition to an exit node that needed some TLC.

Wishing I had an operator manual with general overview in addition to tips and tricks. This is why I created https://github.com/sudomesh/sudowrt-firmware/issues/118 .

So, it looks like the firmware built using the docker "build machine" functions as expected and we are one step closer to a fully automated build process.

paidforby commented 6 years ago

To give an update:

I attempted to build the firmware on a raspberry pi, after changing the image requested in the Dockerfile to arm7/armhf-ubuntu, i was able to run the build process, though it did not appear to produce any binary images. More work needed if to run on arm arch.

I also attempted to run the build process on the communal machine in sudoroom; however, was unable to install docker-ce because there is no installation available for the distro running on that machine (Parrot/JollyRoger).

I am now setting up an old i5 laptop to put on the mesh as build service to be accessed by anyone with a mesh node, per bug 9 (https://github.com/sudomesh/bugs/issues/9).

paidforby commented 6 years ago

I'm happy with the two-liner method I documented as the super-easy way. If anybody doesn't find this sufficient, feel free to reopen this issue.