search

OE4T / meta-tegra

BSP layer for NVIDIA Jetson platforms, based on L4T
MIT License
422 stars 231 forks source link

TK1: L4T R21.6 #57

Closed fkromer closed 6 years ago

fkromer commented 7 years ago

NVIDIA released L4T R21.6 in October 2017. Do you plan to support it?

madisongh commented 7 years ago

I'd love to, but unfortunately I don't have a TK1 dev kit of my own - the 21.5 support was contributed. If someone could help with getting the updates integrated, I'll happily pull them in.

fkromer commented 7 years ago

Ah, I see: PR

luca-nardelli commented 6 years ago

Hi @madisongh, I have a TK1 dev kit and could try to look into contributing the 21.6 support. Right now I'm building standard 21.5 using the rocko release of Poky (and thus I'm working on the rocko branch of meta-tegra), I'll keep you updated.

madisongh commented 6 years ago

Thanks, @luca-nardelli, that would be a big help!

luca-nardelli commented 6 years ago

It seems the kernel changed as well (https://nv-tegra.nvidia.com/gitweb/?p=linux-3.10.git). Since the kernel recipes point to https://github.com/madisongh/linux-tegra, could you create the two branches l4t-r21.6 and patches-l4t-r21.6? If you don't have time to cherry-pick the commits from patches-l4t-21.5 I can do that.

Also, I was having a look at the config.tbz2 archive found in the driver package, which is basically not processed by the recipes. There are some interesting scripts over there that get called by upstart in the standard linux4tegra, for example /etc/init/nv.conf that among other things sets the mmc read ahead queue to 2048 kB instead of the standard 128 kB (in theory it should help when reading files). There's also some similar scripts for systemd that get copied over on the jetson, but as far as I know they don't get used.

Do you think it makes sense to create some kind of tegra-config package that sets these defaults? I believe that if NVIDIA set them there they should be beneficial for the system.

EDIT: I saw the tegra-configs package you made for 28.1, I'll see if I can do something similar for 21.6

madisongh commented 6 years ago

I have pushed the l4t/l4t-r21 branch from NV's repository into the one here, as well as a patches-l4t-r21.6 branch with those few patches rebased. It looks like the new NV branch is a direct successor to the old one, and the patches applied cleanly, so hopefully it will compile OK.

luca-nardelli commented 6 years ago

Kernel compiles successfully with gcc 7.2.0 (rocko). I've started pushing commits here https://github.com/luca-nardelli/meta-tegra/commits/rocko (I'm working directly on the rocko branch, is that ok or should I work on master?).

Image builds and boots =) I still have to check if they are using another version of u-boot as well, right now I'm using the binary provided by NVIDIA in their flashing tools. I've added a tegra-tools recipe that adds tegrastats to the image, and also a tegra-configs recipe where I tried to emulate what you did for the 28.1 version. Since the systemd scripts were not so different from the upstart ones, I've used those. I discarded the firstboot script since it seems to only create symlinks to libraries which are already placed in the correct place by tegra-libraries. I've also enabled USB3.0 by default in the kernel recipe (from extlinux.conf), let me know if that's fine.

What tests do you usually do on the image?

madisongh commented 6 years ago

Great! Rocko is fine to start with. As for testing, I build and boot a core-image-sato image and check

If all these check out OK, that's enough to get started with... although I'm open to suggestions for additional tests, too.

I try to keep the configuration as close to the stock config that NV provides in L4T as I can, but any deviations that folks would find useful are fine.

luca-nardelli commented 6 years ago

Got it! I'll work on the tests.

It makes sense to be close to stock L4T. I'll see if the base upstart script can be used with systemd: the difference is minimal and lies in the configuration of the interactive governor's parameters, but it's better to be close to that. As for USB3, usually I'm always changing that manually, I can try and see if I can make it configurable at a machine/local.conf level (like with a specific MACHINE_FEATURE or similar, suggestions are welcome), does it make sense? Otherwise I can revert the change without issues.

fkromer commented 6 years ago

@madisongh @luca-nardelli For more information about testing of meta-layers the Yocto project QA Master Test Plan can give some inspiration. Especially the Yocto project BSP test plan could be interesting.

luca-nardelli commented 6 years ago

So, I've built the core-image-sato image:

In terms of packages, I wrote recipes for visionworks (1.4.3) and its modules (sfm + tracking) for the K1, along with opencv4tegra. I saw that visionworks is already there for the X1/X2, is it ok to add opencv4tegra as well?

EDIT: I've also noticed that cuda-toolkit packages are missing a COMPATIBLE_MACHINE tag. This causes Yocto to build cuda-toolkit 8 for the K1, which then fails because the other packages have the appropriate tag. I've added the tag to the two recipes (v8 and v6.5)

luca-nardelli commented 6 years ago

@madisongh, I was looking at U-Boot and it changed as well in L4T 21.6 (https://nv-tegra.nvidia.com/gitweb/?p=3rdparty/u-boot.git;a=shortlog;h=refs/heads/l4t/l4t-r21). Could you update your repo (https://github.com/madisongh/u-boot-tegra) in the same way? Thanks!

madisongh commented 6 years ago

OK, I've updated the u-boot repository with the new branches.

I can't see that pastebin post, but I'm not surprised that the X server is presenting a problem. I'm fine with leaving it that way for now, we could maybe mention the issue in the README. The rest sounds good, thanks!

opencv4tegra is obsolete for the X1/X2, you should be able to just use one of the 3.X opencv versions. I already have a bbappend in the layer to enable CUDA and point at the right locations for the libraries it needs. If the TK1 still needs the older opencv4tegra, I'm fine with adding a recipe for it.

Thanks for catching the cuda-toolkit issue!

luca-nardelli commented 6 years ago

Thanks for the repo update, I'll try to build the new u-boot. Sorry for the pastebin link, somehow I made it private instead of unlisted, I've corrected it now https://pastebin.com/kV4vNKP7

As for opencv4tegra for the K1, the main reason was that it supposedly had NEON/CUDA optimizations specifically made by NVIDIA (see http://docs.nvidia.com/gameworks/index.html#technologies/mobile/opencv_known_issues.htm?TocPath=Technologies), do you happen to know if they were ported to opencv 3.X or if opencv 3.X performs better?

The README change sounds good to me.

madisongh commented 6 years ago

For the X problem... I've seen this error before, when trying to use a recent version of X from OE-Core with one of the L4T R24 releases on the TX1/2:

/usr/bin/Xorg: symbol lookup error: /usr/lib/xorg/modules/drivers/nvidia_drv.so: undefined symbol: xf86BlockSIGIO

I think the only way we'll resolve the ABI mismatch is to build an older version of the Xorg server.

OpenCV has some instructions for directly building for the TK1 here, although I don't know how current they really are.

luca-nardelli commented 6 years ago

I see that they mention 3.1 as OpenCV target version. I've compiled opencv countless times (version 2.4.X though, not 3), I'll see if it builds for the K1.

Reading your link, I've noticed this

Note: The 2.4.X OpenCV source does not have the extra modules and code for Tegra that was upstreamed into the 3.X versions of OpenCV. This part of the guide is only for cases where you want to build a vanilla version of OpenCV 2.4.

This makes me think that the optimizations that were advertised for opencv4tegra have been brought to opencv 3. I didn't know that! I'll probably have to create and run a benchmark to actually see if it's real. Anyway, I'll remove the opencv4tegra recipe for the time being.

I went on with working on the port (https://github.com/luca-nardelli/meta-tegra/commits/rocko). Main changes:

Assuming I haven't forgotten anything, I think this should be it for the core packages, @fkromer could you also give it a try?

fkromer commented 6 years ago

@luca-nardelli I would love to. Unfortunately I have no TK1 at home and at work it does not look like this topic gets the required priority to justify it in the foreseeable future.

madisongh commented 6 years ago

That's great, Luca, thanks for all the hard work. I'm OK with pulling these in, and we can just note in the README or on the wiki that testing so far has been limited.

If you could rebase the changes to clean them up, and format the commit messages in the OE style, that would be great. If you don't have time for that, I'll take care of the cleanup when I pull them, which I probably won't be able to get to until this weekend.

luca-nardelli commented 6 years ago

I've been playing a bit with trying to compile opencv 3 today.

With GCC 7.2 I'm getting a ton of errors: at first they were all cuda-related, so I thought that was because CUDA 6.5 doesn't properly support C++11 and gcc 7.2 is c++14 by default I think, then I put opencv in c++98 mode (ENABLE_CXX11=OFF and CXXFLAGS += "-std=c++98") and I got a bunch of other errors, some of them related to Eigen, which says that some of its datatypes changed in GCC 7.1. If I try to build with my port of GCC 4.9 it builds fine though.

@madisongh have you succesfully built opencv 3.3 for the tx1/tx2? What GCC were you using?

EDIT: Just saw your reply, should I squash the commits when rebasing? I'll remove some of the tests I've made (e.g. USB3). I'll look into formatting them as well and let you know!

fkromer commented 6 years ago

@luca-nardelli Probably helpful for you: For automation of tests of that kind one can use e.g. the avocado-framework (successor of autotest which merges or has already merged ltp tests into thei test repository avocado-misc-tests). The test case repository for avocado contains tests to automate proper functioning of tools, here gcc: avocado-misc-tests/toolchain/gcc.py . (It is also possible to test several combinations of configurations for a single test automatically, in your actual case probably versions of tools.)

madisongh commented 6 years ago

@luca-nardelli CUDA 8 on the TX1/TX2 requires gcc 5.x, so that's the version I use when building opencv 3.x for those platforms.

If you can squash any fixups you needed to make, that would help cut down on clutter in the commit history. Thanks!

@fkromer OE has some built-in automated testing support, too, and I'd love to have at least a semi-automated test setup, but as I and the other contributors are all doing this in our spare time, it's been difficult to devote the time, effort, and cash needed to get it set up.

luca-nardelli commented 6 years ago

Got it, I thought that since you had a rocko branch you were working with 6.3 or 7.2 given that only those two compilers are available in the rocko branch. It appears that I was completely on the wrong branch for all of the Jetsons (for what concerns CUDA development).

I'll work on the cleanups and on the commit styles.

How do we proceed with the other branches? I think that if someone wants to to CUDA development on the TK1 with this layer they should use the Jethro branch, should I bring the modifications over there after cleaning up?

madisongh commented 6 years ago

I do build images without CUDA support using the latest available compilers, too. Unfortunately CUDA is always way behind with gcc versions. I thought I had a note on the wiki about this problem and where to get recipes for older toolchains. We could update that with the CUDA 6.5 info.

I don't like recommending sticking with really old OE-Core/Yocto releases... it's hard enough maintaining two recent releases plus keeping up with master for the next release. I'm not sure there's a good solution for this, though, if NVIDIA never issues significant version updates for the TK1.

I'll definitely merge this into master for the future. And I suppose that since 21.6 is a very minor change from 21.5, we could back-port to older branches without too much hassle.

luca-nardelli commented 6 years ago

I didn't find links to older toolchains on the wiki. You were probably referring to this link right? Here they mention that copying the recipes worked, but there is no link as well.

For the TK1, we could add the meta-compilerports layer I've started to work on, although I should port 4.8 as well since that's the required version for CUDA 6.5. There's also meta-linaro but their compilers for the rocko branch start from version 4.9. Unfortunately I fear that support for the TK1 is practically non-existent now, in terms of CUDA toolchain updates.

You're perfectly right when not recommending old Yocto releases, I've been stuck with jethro for so long and I just recently had some time to start looking at the new releases.

By the way, I've made an attempt at cleaning up the commit history here. It's my first attempt following the style, so please let me know if something could be made better!

madisongh commented 6 years ago

Looks great, thanks! Go ahead and open a pull request and I'll get them integrated.

On the toolchain stuff, I'll add some links to the wiki page. IIRC from the original discussion about 21.5, gcc 4.9 was compatible enough with CUDA 6.5 that it could be used without major problems.

madisongh commented 6 years ago

Fixed by #60.

fkromer commented 6 years ago

@madisongh "OE has some built-in automated testing support, too, and I'd love to have at least a semi-automated test setup, but as I and the other contributors are all doing this in our spare time, it's been difficult to devote the time, effort, and cash needed to get it set up."

I know. In my spare time testing is way not my favorite activity either :smiley: I just answered because @luca-nardelli asked for info...

madisongh commented 6 years ago

FYI, I got my hands on a TK1 dev kit and have fixed the X server issue (by porting in an older xserver-xorg recipe). I've done a bit of cleanup on the 21.6 integration recipes to align better with the TX1/TX2. And I've added support for direct flashing of the TK1.

I can get my test movie to play on my HDMI display using the nvhdmioverlaysink gstreamer plugin, with audio routed out the speaker jack, although it looks like something's not quite right there.

estatz commented 4 years ago

@madisongh Sorry to dredge up an old issue about even older hardware, but i'm trying to build an image with CUDA support for the TK1 and found my way to some of the important discussion in this thread so I figured others might find their way here if they were attempting to do the same thing. There are some mentions in this thread about using rocko with GCC 4.9, and some mentions on the wiki Compatibility-notes about using GCC 5 all the way up to warrior. I've also found the wiki pages (and link to slides from Yocto Summit 2019 at the bottom of the wiki) about various options for using a GCC 7 toolchain with warrior+. Is the GCC 7 discussion the best place to start if I wanted a template for trying to make GCC 5 + CUDA 6.5 work with rocko or newer? Which yocto release was the latest where CUDA was tested on the TK1? If the latest CUDA testing on TK1 happened on something newer than jethro, am I missing any additional breadcrumbs for corresponding toolchain setup? Thank you!

madisongh commented 4 years ago

@estatz It's been a while... I don't remember whether or not I ever actually verified building and running any CUDA apps for the TK1 back then, and was relying on others to test for me. I think there were some hacks and/or caveats involved, but I don't remember what they are, and everything I can find today indicates that you really need GCC 4.x. Sorry I can't be more help... maybe someone else on this thread has some insights they can share.