search

rodizio1 / EZ-WifiBroadcast

Affordable Digital HD Video Transmission made easy!
GNU General Public License v2.0
817 stars 200 forks source link

Image-builder produces stable 2.0 image #166

Closed RespawnDespair closed 5 years ago

RespawnDespair commented 5 years ago

Fix all the known problems in the image builder so we have a stable and usable 1.6 image.

https://github.com/RespawnDespair/wifibroadcast-image-builder/projects/1

Yes21 commented 5 years ago

5.2.20 appears incompatible with the latest kernel from github.

That's why I'm now testing with 5.3.4 ...

RespawnDespair commented 5 years ago

@careyer yes, good idea, i'll implement it in the builder. @Yes21 Are the timestamps from the kernelx.img recent or are they from the previous run? Just to clear any possible confusion. If 5.3.4 works i'll update the github to that branch

Yes21 commented 5 years ago

Are the timestamps from the kernelx.img recent or are they from the previous run?

They are later than the last build.

The build is now finished !

Will now have a walk with the dog, and then burn my latest build with 5.3.4 ...

RespawnDespair commented 5 years ago

Please let me know how it goes, very interested!

careyer commented 5 years ago

Okay, new insight..... as an AirPi I use a Raspberry Pi A+ (this is part of my test setup)... it uses the same kernel as a Pi0. It does not work with the BuildImage.

The BuildImage however works as a AirPi when running on a Pi3B

Yes21 commented 5 years ago

I can confirm that the image doesn't work on my Pi zero, but works on my Pi 2B and Pi 3B.

careyer commented 5 years ago

@Yes21 : high five! =)

RespawnDespair commented 5 years ago

Weird, it used to work on the zero as well. Does it boot at all on the zero or a+?

careyer commented 5 years ago

Yes it boots.... but it stops saying:

MAybe Yes21 can share what it sais on tty1 (Alt+F1) ... my TV is still occupied by family :-)

Yes21 commented 5 years ago

I'm sad to say that my latest image with rtl8812au v5.3.4 doesn't recognize 8812au and 8814au !! It's seems that the driver is not installed.

With v5.1.5, I can see the 8812au kernel module in :

/lib/modules/4.14.78-v7+/kernel/drivers/net/wireless/realtek/rtl8812au/8812au.ko

But with v5.3.4 I can't see any 8812au module ...

This driver is giving me headache :-(

Yes21 commented 5 years ago 
pi@wbc(rw):~$ ls /lib/modules/
4.14.71+/    4.14.71-v7+/ 4.14.78-v7+/ 
pi@wbc(rw):~$ uname -r
4.14.78-v7+

We are now running 4.14.78, but there's no 4.14.78+ ! Only 4.14.78-v7+ ...

Perhaps it will be a good idea to fix the kernel version we want to build/use ?

RespawnDespair commented 5 years ago

Perhaps the problem is i only run mod install after the v7 build. Maybe it needs to be run two times, once after v6 compile and once after v7...

Yes21 commented 5 years ago

@RespawnDespair : Do you know how I can force the raspbian version to a fix kernel version ? I want to try to build a 4.14.66 (or sooner) kernel, because I'm sure that the rtl8812au v5.2.20 was compiling perfectly with this kernel, when I did test latest DroneBridge a few days ago.

careyer commented 5 years ago

@RespawnDespair : I guess you are on the right track.... I think you have to run mod install after both kernels... same will be true for the v4l2loopback kernel module

RespawnDespair commented 5 years ago

@careyer Yeah, the v42loopback is next on the list. First get the basics working. I'm modifying the scripts, but i'm not sure i'll get it built and tested tonight. I'll post here once i have the fixes checked in, maybe you can try to build and test as well...

@Yes21 you could modify the git clone command that gets the kernel to point to the specific kernel you want. I'm not sure if there are branches you can refer or if you have to use the sha-1 commit...

Yes21 commented 5 years ago

Thanks to @dposea I've understood that before compiling the rtl8812au driver we have to edit the Makefile :

CONFIG_PLATFORM_I386_PC to n 
CONFIG_PLATFORM_ARM_RPI to y

But that's not enough ...

RespawnDespair commented 5 years ago

Looking at the makefile i also see this section:

########################## WIFI IC ############################
CONFIG_MULTIDRV = n
CONFIG_RTL8188E = n
CONFIG_RTL8812A = y
CONFIG_RTL8821A = y
CONFIG_RTL8192E = n
CONFIG_RTL8723B = n
CONFIG_RTL8814A = n
CONFIG_RTL8723C = n
CONFIG_RTL8188F = n
CONFIG_RTL8822B = n
CONFIG_RTL8723D = n
CONFIG_RTL8821C = n

Should the CONFIG_RTL8814A be set to y as well? I don't touch the CONFIG_PLATFORM_ARM_RPI setting yet either... I don't have an 8812 or 8814 for testing, perhaps you can build with the CONFIG_PLATFORM_ARM_RPI to y and the CONFIG_RTL8814A to y as well?

Yes21 commented 5 years ago

Yes, I will try it tomorrow afternoon with v5.1.5, with a modified Makefile. But I'm almost sure there's also a problem with kernel version, which is changing very fast ! I couldn't yet find how to clone a precise version of raspbian. I can't choose better than 4.14.y !

Have a good night.

Yes21 commented 5 years ago

Last week the script was building a 4.14.54 kernel, and today a 4.14.78 ! How could we build a stable image, if the kernel version is changing each time we are building a new image ?

Just a little thought before going to bed ... ;-)

Yes21 commented 5 years ago

Thanks to @seeul8er (one more time) I found the link to clone 4.14.66 : https://github.com/raspberrypi/linux/tree/c5f7d3c4daf14ba3717fcc1497854d8c365bd742. I also saw he's using rtl8812au v5.2.20 with a modified Makefile, like we spoke a few hours ago. I will use these 2 things for my next build.

RespawnDespair commented 5 years ago

That is the sha-1 method i mentioned earlier. Can you point me to the modified makefile? Interest only, not going to implement it right away. I did do fixes on the image-builder, you might want to do a git-pull again.

Yes21 commented 5 years ago

Thanks to @seeul8er (one more time) I found the link to clone 4.14.66 : https://github.com/raspberrypi/linux/tree/c5f7d3c4daf14ba3717fcc1497854d8c365bd742.

Doesn't work ...

Yes21 commented 5 years ago

That is the sha-1 method i mentioned earlier.

I don't know this method. I will try to understand what it is ...

Can you point me to the modified makefile?

https://github.com/aircrack-ng/rtl8812au/blob/v5.2.20/Makefile

RespawnDespair commented 5 years ago

I don't know this method. I will try to understand what it is ...

That's the link you posted to the linux repo, the part after /tree/ is the SHA-1 key (c5f7d3c4daf14ba3717fcc1497854d8c365bd742 in this case). I had never used it before, but i know it was possible, basically it allows you the get the source tree at the time of the commit the SHA-1 refers to. Every commit in Github has such a key.

The makefile is basically the same as 5.1.5 except the 8814au is also y in there, there will offcourse be other changes in the source.

Yes21 commented 5 years ago

That's the link you posted to the linux repo, the part after /tree/ is the SHA-1 key (c5f7d3c4daf14ba3717fcc1497854d8c365bd742 in this case).

Ok, I understand. I've searched 2 hours long on the net, but I can't find a working solution to clone this version. Before I find a solution, I will download the zip file and try to build the image like this.

I'm really curious to see if rtl8812au 5.2.20 can compile with kernel 4.14.66.

RespawnDespair commented 5 years ago

See the accepted answer here:

https://stackoverflow.com/questions/3489173/how-to-clone-git-repository-with-specific-revision-changeset

That explains how to clone a specific changeset, just use the sha-1 from your earlier post. Hope it helps!

Yes21 commented 5 years ago

Before I find a solution, I will download the zip file and try to build the image like this.

No, that's not working : no kernel*.img ! Damned ! I don't understand why this driver doesn't want to compile ...

I'm quite sure it was compiling with the "manual" image building process of DroneBridge. I will redo this process tomorrow to be sure ...

It's along way !

RespawnDespair commented 5 years ago

Don't get to frustrated, we'll tackle this problem. Perhaps let it lie for a couple of days. I will have some time this weekend as well, perhaps i could try as well.

careyer commented 5 years ago

It is late - don't worry your heads about it and get some time with your family and beloved ones. Tomorow comes another day. Let me know if I can help in testing anything as my build environment also works now. Good night!

Yes21 commented 5 years ago

@RespawnDespair @careyer : Yes, tomorrow will always be now. Thanks for your kind words. Have a good night.

Yes21 commented 5 years ago

I'm quite sure it was compiling with the "manual" image building process of DroneBridge. I will redo this process tomorrow to be sure ...

Last idea for going to bed : I will rather "replay" your script step by step, to be able to see compilation errors of the driver.

Yes21 commented 5 years ago

Hello, that's a good day and the sun shine, and ...

I've just succeed in compiling the kernel and the rtl8812au v5.2.20 driver !!

Here is the compile error :

drivers/net/wireless/realtek/rtl8812au/Makefile:305: /home/yves/ez-wbc/wifibroadcast-image-builder/work/02-Kernel/linux/hal/phydm/phydm.mk: Aucun fichier ou dossier de ce type

The hal dir is not in /linux/, but in /rtl8812au/. So I've just copied rtl8812au/hal in /linux/hal. And it worked !!

I got the kernel :

root@PC-Canopeo:~/ez-wbc/wifibroadcast-image-builder/work/02-Kernel/linux# ls -al arch/arm/boot/
total 15044
drwxr-xr-x   5 root root     4096 nov.   3 14:23 .
drwxr-xr-x 100 root root     4096 août  22 07:46 ..
drwxr-xr-x   2 root root     4096 août  22 07:46 bootp
drwxr-xr-x   2 root root     4096 nov.   3 14:23 compressed
drwxr-xr-x   2 root root    73728 nov.   2 20:45 dts
-rw-r--r--   1 root root       40 août  22 07:46 .gitignore
-rwxr-xr-x   1 root root 13168148 nov.   3 14:23 Image
-rw-r--r--   1 root root      109 nov.   3 14:23 .Image.cmd
-rw-r--r--   1 root root     1648 août  22 07:46 install.sh
-rw-r--r--   1 root root     2816 août  22 07:46 Makefile
-rwxr-xr-x   1 root root  4758176 nov.   3 14:23 zImage
-rw-r--r--   1 root root      136 nov.   3 14:23 .zImage.cmd

and the 88XXau module :

root@PC-Canopeo:~/ez-wbc/wifibroadcast-image-builder/work/02-Kernel/linux# ls -al drivers/net/wireless/realtek/rtl8812au/
total 5932
drwxr-xr-x 8 root root    4096 nov.   3 14:23 .
drwxr-xr-x 6 root root    4096 nov.   3 14:23 ..
-rw-r--r-- 1 root root 2923592 nov.   3 14:23 88XXau.ko
-rw-r--r-- 1 root root     290 nov.   3 14:23 .88XXau.ko.cmd
-rw-r--r-- 1 root root   15306 nov.   3 14:23 88XXau.mod.c
-rw-r--r-- 1 root root   20012 nov.   3 14:23 88XXau.mod.o
-rw-r--r-- 1 root root   21752 nov.   3 14:23 .88XXau.mod.o.cmd
-rw-r--r-- 1 root root 2904340 nov.   3 14:23 88XXau.o
-rw-r--r-- 1 root root   10490 nov.   3 14:23 .88XXau.o.cmd
-rw-r--r-- 1 root root       8 nov.   3 14:23 built-in.o
-rw-r--r-- 1 root root     195 nov.   3 14:23 .built-in.o.cmd
drwxr-xr-x 3 root root    4096 nov.   3 14:23 core
-rw-r--r-- 1 root root     326 nov.   2 12:55 dkms.conf
-rwxr-xr-x 1 root root     510 nov.   2 12:55 dkms-install.sh
-rwxr-xr-x 1 root root     506 nov.   2 12:55 dkms-remove.sh
drwxr-xr-x 8 root root    4096 nov.   2 12:55 .git
-rw-r--r-- 1 root root      81 nov.   2 12:55 .gitignore
drwxr-xr-x 9 root root    4096 nov.   3 14:23 hal
drwxr-xr-x 5 root root   12288 nov.   2 12:55 include
-rw-r--r-- 1 root root     109 nov.   2 12:55 Kconfig
-rw-r--r-- 1 root root   18092 nov.   2 12:55 LICENSE
-rwxr-xr-x 1 root root   61091 nov.   2 12:58 Makefile
-rw-r--r-- 1 root root      56 nov.   3 14:23 modules.order
drwxr-xr-x 3 root root    4096 nov.   3 14:23 os_dep
drwxr-xr-x 2 root root    4096 nov.   3 14:23 platform
-rw-r--r-- 1 root root    2723 nov.   2 12:55 README.md
-rw-r--r-- 1 root root    2322 nov.   2 12:55 Realtek_Changelog.txt

I know my solution is not very clean. But I'm sure you will find a smarter way to include it in your script ! (perhaps just creating a symlink ?)

I did this test with 4.14.66 kernel and 2018-06-27-raspbian-stretch-lite, but I think it should work with other kernel. I really don't know if we have to stick whith 4.14.66. Perhaps yes, because the kernel patchs your are using are for 4.14.66.

(I didn't yet find a clean way to upload 4.14.66 kernel sources ...)

RespawnDespair commented 5 years ago

Very good news, i still like to use the latest mainline kernel, but at least now we know it is possible! I have just recieved my Pi3B+ and it's running EZWFB using the last image i made without any apparent problems. Progress i'd say! I will try to check the RTL88xx drivers in my image and try to adjust the build process!

Yes21 commented 5 years ago

@RespawnDespair : I also propose to put the modified rtl8812au Makefile in "stages/02-Kernel/FILES/", and modify the "stages/02-Kernel/00-run.sh" to copy the modified Makefile after cloning the driver (-b v5.2.20).

Yes21 commented 5 years ago

@RespawnDespair

Progress i'd say!

YES that's really a very good day ;-))

careyer commented 5 years ago

AWESOME WORK! I told you this would be a good day just yesterday. Sometimes you have to take a step back and free up your mind to see the big picture more clearly. Keep it on... The both of you, you are an awesome team.

Von meinem iPhone gesendet

Am 03.11.2018 um 15:21 schrieb Yes21 notifications@github.com:

@RespawnDespair

Progress i'd say!

YES that's really a very good day ;-))

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Yes21 commented 5 years ago

@RespawnDespair : And now, the way to clone the 4.14.66 kernel sources :


git clone --depth=100 https://github.com/raspberrypi/linux/
cd linux/
git checkout c5f7d3c4daf14ba3717fcc1497854d8c365bd742
Yes21 commented 5 years ago

@RespawnDespair

Perhaps the problem is i only run mod install after the v7 build. Maybe it needs to be run two times, once after v6 compile and once after v7...

Don't forget to fix that.

RespawnDespair commented 5 years ago

@Yes21 I have made several modifications to the builder. I do not use your patched makefile (yet), but i have written a small patch in the main patch file that sets the basepath. The kernel compiled correctly (it is still the 4.14y kernel, so the .78) but with the 5.2.20 version of the driver. What should i look for to know 8814 support is now in there?

Yes21 commented 5 years ago

@RespawnDespair I think you should at least have this line (in the mounted image) :

/lib/modules/4.14.78-v7+/kernel/drivers/net/wireless/realtek/rtl8812au/8812au.ko and perhaps idem for 8814.ko ...

And also that in the kernel sources :

root@PC-Canopeo:~/ez-wbc/wifibroadcast-image-builder/work/02-Kernel/linux# ls -al drivers/net/wireless/realtek/rtl8812au/
total 5932
drwxr-xr-x 8 root root    4096 nov.   3 14:23 .
drwxr-xr-x 6 root root    4096 nov.   3 14:23 ..
-rw-r--r-- 1 root root 2923592 nov.   3 14:23 88XXau.ko
...

But my working dir is now too dirty, and will restart with a new clone of your latest modifications.

RespawnDespair commented 5 years ago

Well, that file is there, so i'll count it a success then. After talking with @rodizio1 we've decided to fix the kernel at 4.14.71, that's the kernel supplied in the latest Raspbian Lite we are using as the base. I am modifying the scripts and testing them now.

Yes21 commented 5 years ago

I think that's a very good idea to fix the kernel version.

RespawnDespair commented 5 years ago

The latest image i built with the latest code in image-builder appears to work. The basic functionality is there. Does anybody else have a latest image? Just curious to the results.

Yes21 commented 5 years ago

I've just finished building it. I'm now burning my 2 sd cards. Will test it soon ...

Yes21 commented 5 years ago

And we have now a new tester : https://www.rcgroups.com/forums/showpost.php?p=40435283&postcount=10

RespawnDespair commented 5 years ago

https://drive.google.com/open?id=1BvW3fG6TUBoGa5687N-RFzpQUuoQPOj5 Is the latest image from my local build, anybody using the latest builder should produce the same...

Yes21 commented 5 years ago

Awesome !! My first test is very positive !! I think we got it. Can you wait a few more time for my complete revue ?

RespawnDespair commented 5 years ago

Off course! Very positive! Take your time, if there are issues report them separately, i'll try to fix any remaining issues. Thank you for your patience and time!

Yes21 commented 5 years ago

And now, ... my first very positive test !!

That's wonderfull !! It's the first time I can see so many things working well with my setup !!

We did very good work this weekend ... Many thanks to you, for all the time and energy you did give to these long steps !

But this is not the end ...

careyer commented 5 years ago

Totally agree... You did some amazing work! You have all the reason to be proud!!! My image build also finished... however my IMAGE.img and the one Jelle linked above give me different MD5 and SHA1 checksums. Jelle, have you changed anything after I started my build?

UPDATE: Decided to go with the downloaded image from Jelle.

GroundPi (Pi3B)

AirPi (PiA+)

All seems good but there seems to be a serious performance problem at least with PiA+. The things marked with (*) work nicely with the genuine 1.6rc6 image. Otherwise! Great JOB! And what a neat logo =)