ShabbyX / RTAI

(NO LONGER MAINTAINED) Clone of RTAI from https://www.rtai.org
28 stars 17 forks source link

rtai-config only supports a single kernel #16

Closed zultron closed 9 years ago

zultron commented 10 years ago

rtai-config (as handed down from upstream) contains hard-coded parameters such as kernel version. As such, even when multiple sets of RTAI kernel headers are installed, rtai-config can only be used to build against one of them.

This is a bigger problem when packaging, since rtai-config is part of the librtai-dev package, of which only one instance may be installed on a system.

This problem might only affect weird folk like me, who want to build and test against a variety of kernel versions, but I'd like to hear if others are affected, and would like to hear opinions on how this is best solved.

I see two solutions for packaging. One is to install an rtai-config script with hard-coded parameters, as is done now, but in the same package with the RTAI modules and with a unique name (e.g. rtai-config-$(uname -r)). The original rtai-config script could be replaced with a stub script that uses overrideable heuristics to pick a reasonable default set of kernel headers, and then executes the matching rtai-config-<kver> script.

Another is to modify the rtai-config script, removing those hard-coded parameters that may vary with kernel headers, and automatically generating them as above, by picking a reasonable default set of headers and extracting the correct values.

Discussion appreciated.

zultron commented 10 years ago

Previous discussion on the 'Rtai' list:

Shahbaz suggests something like the first idea (I must've stolen it from here): http://mail.rtai.org/pipermail/rtai/2013-June/025583.html

Similar method used by Richard Jackson: http://mail.rtai.org/pipermail/rtai/2013-June/025579.html

Start of thread, relevant section near the end: https://mail.rtai.org/pipermail/rtai/2013-June/025578.html

zultron commented 10 years ago

The advantage of adding a stub rtai-config that picks a reasonable default set of headers is it's compatible with upstream behavior, and should therefore surprise users less.

Heuristics I'd consider, depending on others' opinions and difficulty in scripting, with rationale:

Some of these are a bit involved to script, and may quickly hit the point of diminishing returns, but the first two are essential, and the next two could be replaced by erroring out, but after printing a list of configured headers found.

ShabbyX commented 10 years ago

@zultron , first of all, I've done something like the following in one my projects: In the configuration of the project (let's call it P), you select a suffix. Then every output of the project would have that suffix. For example libP$(suffix).a, libP$(suffix).so, P-config$(suffix) etc. So once configured, you can do P-config-1.2 or P-config-dev or whatever to easily build and configure against the different versions. You could make a symlink named P-config, but that could only point to one of them (the most recent installation for example).

Perhaps we could add an option to RTAI's build to add the kernel version as suffix?

Regarding automatic detection of the kernel, I would be very cautious. First of all, if it fails to find the one user wants, it would cause some frustration. I would personally prefer the script to refuse to work until I give the --kernel-version (in case there are more than one installations, which itself is hard to figure out). That way, I don't accidentally use the wrong configuration.

Perhaps the rtai-config script could look for rtai-config-* that are available and offer a choice to the user (if --kernel-version is not given, and there are more than one of them), instead of choosing one automatically? This way, everything stays the same (hardcoded) within rtai-config-* files, while rtai-config would simply be an interface to choose one of them.

If you feel like implementing this, it would be great. :)

zultron commented 10 years ago

I'm having trouble understanding the relation of your first paragraph to the rtai-config problem, probably because I'm short on sleep and don't have much experience with RTAI. Can you spell it out for my foggy brain? In Machinekit, I've set up the build and run-time scripts so that kmodules go into their own directory (whose path contains $(uname -r)) separate from the RTAI modules. Is that related to the problem your suffix scheme solves?

About kernel detection, I trust your instinct about failure and frustration. So am I right that we agree:

The menu idea is great.

My thinking behind the second heuristic:

In most RTAI environments I've seen, there's typically just one RTAI kernel + headers + RTAI modules, and that's the running kernel. The users' work flow involves building against the running kernel, installing the kmodules and running the application. There's no other choice of kernel and headers.

IIRC, the LinuxCNC build system simply runs rtai-config without regard to which kernel it configures; this is a de-facto (rather arbitrary) default. That works most of the time, but I bet that's caused some consternation in the past!

In the case where there's no other (detected) choice, I think it makes sense to simply pick the only available option as the default when no --kernel-version is supplied. Otherwise, I advocate for printing a list and bombing out, and optionally adding a -i/--interactive or similar option to enable simple manual selection from the keyboard.

ShabbyX commented 10 years ago

Regarding my suffix scheme, kind of like what you say, but used in more situations. For example, liblxrt.so could also become liblxrt-$(uname -r).so. And the rtai-config-$(uname -r) script give --lxrt-ldflags would also give -llxrt-$(uname -r) (hardcoded values for uname -r, or actually the kernel version RTAI is built against). In short, anything that can get mistaken (e.g. the kernel module, or the shared object) have the suffix.

Ok, I agree with everything you just said. In case of no --kernel-version and a running RTAI kernel, choosing the running kernel seems like the perfect choice. Also, if there is only a single RTAI installation detected, of course choosing that automatically is desirable.

So, would you take the trouble of implementing this?

zultron commented 10 years ago

I get the suffix scheme for /usr/bin/rtai-config-<kver> script; that matches my thinking exactly.

I still don't get the uniquely-named liblxrt.so. Do the userland libs need to vary with each kernel, or would a single liblxrt.so (installed by the librtai1 Debian package) work for any RTAI kernel?

My packaging is based on the upstream Debian packaging, which builds the following (kernel-independent) packages:

It appears the intent of the Debian packaging author, Roland Stigge, is that userland libs and dev headers are shared. This makes it easy (ha ha) to build a kernel package with RTAI kmodules included. I think if the RTAI kernel package included a /usr/bin/rtai-config-<kver> and we sorted out the universal /usr/bin/rtai-config script, the packages would be in pretty good shape, and it would be a lot easier for folks to install and use RTAI.

I'm still trying to understand what to do here, so I can't answer the question yet of whether/when I'll implement. A need for a per-kernel uniquely-named liblxrt-<kver>.so would require the kernel package be taught to build and install the userland lib, which would be a major restructuring in my packages and frankly something I would not want to consider.

The rtai-config changes sound pretty trivial and I'd be up for that once I get time; ATM the big demand is for ARM-architecture Machinekit and (matching) Xenomai kernel packages. I'm interested in RTAI packaging because it's something both the LinuxCNC and Machinekit projects would benefit from, and because unifying the packaging of Xenomai and RTAI kernels for x86 and ARM architectures in a single easy-to-maintain package makes life beautiful. ;)

ShabbyX commented 10 years ago

Regarding the .so file: In my original scheme, the suffix represents actual different configurations, so libsomething-suffix1.so would actually be different from libsomething-suffix2.so. If different builds of RTAI for different kernels have different configurations too (i.e. differences other than the kernel it's based on), then having the same name for the shared objects (and having them both in the loader path) may cause problems. If the different builds of RTAI have the same configuration, then certainly this is not an issue. Building against a static library of course has no problem.

Ok, I just read more of your post. Adding the suffix to the .so would be useful if someone is manually building RTAI and installing them for different kernels. On the other hand, if they are installing from a package, the builder of the package can of course assure the same configuration, but this is risky business. Another note, the same things I said apply to the static library too (possible different configs for different kernels), but that is easily managed from the rtai-config-<kver> file given that liblxrt.a for each kernel is installed in a different path. I guess this adds to the trouble.

Just to summarize, rtai-config-<kver> should give out correct values for header includes paths, preprocessor definitions and linker path. I'm not sure if that's possible with a unique user-space installation and multiple kernel module installations (each having a (possibly different) rtai-config-<kver>)

Ok I don't know how the packages are built (0 experience in debian packaging), and I trust you that installing both the kernel module and the corresponding library would be a hassle1. So first let's try to figure out if this is a real problem or not. Then we could think about how to solve it.

1 So it's possible to download the source and build and install the modules all from the debian package? Nice! Why not then just build the whole RTAI and install, rather than only the kernel modules?

zultron commented 10 years ago

Hi Shahbaz,

Yeah, I've become more familiar than I ever wanted with both Red Hat and Debian packaging, and it's a characteristic of both distro that packages are built with particular arguments to ./configure, and only one version of a package can be installed at a time. Those who want a different configuration can either rebuild the package with different arguments and replace the upstream version, or if they want both simultaneously, they can rework the package with a new name and resolve file conflicts, perhaps using a scheme like yours. Of course this is acceptable to most users, but folks like you (and me, for numerous other packages) who have more specific requirements have to suffer. I get the idea that the Gentoo packaging addresses this problem by allowing the user to specify build params and rebuild all from the packaging system; Alec would know more.

Because of this limitation, the upstream userland RTAI packaging chooses a default. In this case (aside from install paths): --disable-leds --disable-rtailab --enable-comedi-lxrt --enable-comedi-lock. Does that look reasonable for most users?

The kernel packages built against these RTAI packages could contain an ABI name that the RTAI libs could check for, but that would have to come from the RTAI project. The packages could be linked through 'Provides:' and 'Depends:', but that doesn't guarantee the running kernel matches the libs. I haven't thought carefully about how your library naming scheme would work in a packaging context, but it would be a deviation from the usual practice.

In the end, I'm tempted just to ignore the problem and build packages. If it's an issue, we might look to the LinuxCNC RTAI packages, which make these same assumptions and have been in wide use for several years.

As for the rtai-config-<kver>, that script would come in the linux-headers package, and should uniquely match the kernel. Debian kernel versions look like 3.2.0-4-amd64 (vanilla) or 3.8-1mk-xenomai.beaglebone-omap. The 4 and 1mk are the kernel ABI name, which should be bumped whenever the ABI changes. Anything installed by the rtai/librtai1/librtai-dev packages must be kernel-independent, so nothing that includes <kver>. I hope this satisfies your concern about unique user-space installation and multiple kernel module installations?

And to address your footnote, the rtai-source package does an unusual (and kind of disgusting) thing (not my invention, but I also don't see a better way that doesn't involve lots of work). It includes the linux-hal patches and also a tarball with enough of the RTAI distribution to build kernel modules. That package is not normally needed except to build an initial kernel source package, which must include those artifacts in order to rebuild itself. IMO, the kernel package should only be responsible for building kernels, and user-space things need to be packaged separately; we see this in e.g. the linux-tools package, which build perf and other userspace things that complement the kernel.

ShabbyX commented 10 years ago

Hi!

I'm on a sudden vacation now, but I'll think about these. I'm afraid not knowing about the RTAI packaging, I really shouldn't argue with you ;)

My last question about the multiple kernel, single user installation: forget about packaging for a minute. Say you configure RTAI for 3.4, write an application and have it link to shared object of RTAI. Note that the configuration of RTAI is written to a header file and the corresponding effects, if any, is compiled into the application. Now say you change to 3.10, reconfigure RTAI with an arbitrarily different configuration and install it over the previous. The question is, would the application compiled with the previous RTAI just work without the need to recompile?

Assuming the configuration is the same (the one you suggested is sane by the way. Also I suggest a much higher number of objects than 150, and enable RTDM too), then I agree that multiple kernel space, single user space should be fine.

Shahbaz

zultron commented 10 years ago

The only shared object I see RTAI produces is liblxrt.so.1.0.0, does that sound right? And the generated RTAI header file is rtai_config.h?

I believe that as long as the options to ./configure stay the same, and the RTAI codebase hasn't changed significantly (if it does, the .1.0.0 ABI version should be bumped), the .so library ABI should remain the same. I don't see anything in rtai_config.h that looks tied to the kernel version.

If the options to ./configure change, then it could change the ABI. I imagine that could be problematic esp. if a feature used by the application was disabled.

RTAI applications are special, of course, since part of the application may be compiled as kernel modules, as is the case in LinuxCNC. Kernel modules build for Linux 3.4 can't be expected to work properly under a 3.10 kernel, since even if the kernel space RTAI ABI hasn't changed, the Linux ABI may have.

At least earlier RTAI kernels provided by the LinuxCNC project have had kernel symbol versioning turned off, which allows kmodules built for one kernel to be inserted into a different kernel. IMO this is a dangerous thing to do, and my kernels (and upstream distro kernels) all enable kernel symbol versioning. This ensures that kmodules may only be inserted in a kernel with the same ABI.

So to sum up, if you configure and build RTAI for Linux 3.4, and install the userland bits (incl. shared libs and headers), and then configure and rebuild for Linux 3.10:

For userland bits, I think that there's no reason to reinstall if ./configure args haven't changed. If they have changed, RTAI userland ought to be reinstalled, and perhaps must be reinstalled if RTAI features were removed. The same holds for the userspace portion of an RTAI application. So when packaging, we should be careful about changing args to ./configure.

For kernel-space artefacts, whenever the kernel is changed, the application's kmodules need to be rebuilt against the new kernel headers to ensure any kernel ABI changes are picked up.

Does this answer the question? (Disclaimer: I don't know a whole lot about RTAI, and know a bit about kernels and ABIs but certainly not everything!)

Thanks for the tip about raising object numbers and enabling RTDM. When I get close to showing off packages, I'll certainly ask your help to review the configuration.

ShabbyX commented 10 years ago

Comment to your disclaimer: me neither!

Ok, I'm convinced. So long as RTAI is configured suitable for the general user (as opposed to people with specific requirements, which would compile from source anyway), I don't see a reason why the configuration would change between different kernel users.

Since you say that rtai_config.h is not kernel version dependent, then your original scheme (one kernel-space installation for each kernel version and a single user-space installation) should work alright.

I would create an issue right now and send a link to the mailing list, asking people what configuration they need in particular. I don't think there would be many (if any at all) conflicting requirements, so hopefully with people's feedback we'd get a configuration that satisfies everyone.

zultron commented 10 years ago

I spent a little more time on these RTAI packages, and came up with a start on this issue.

The following patch teaches RTAI to install the current style of rtai-config script into rtai-config-<utsname>, which can be run simply as rtai-config-$(uname -r).

The stub rtai-config looks for executables in $PATH called rtai-config*. In list mode, rtai-config -l, a list of utsnames is printed. Otherwise, rtai-config -v <utsname> or rtai-config -d <headers-directory> should be sufficient to find the right kernel-specific script, and then the usual arguments apply. For example, rtai-config -d <headers-directory> --linux-dir should return the same <headers-directory>.

This is still a WIP while I look into updating Machinekit's autoconf configuration to manage multiple RTAI kernels (and I'm going on vacation myself!). In the meantime, I'm interested in feedback. Thanks!

https://github.com/zultron/rtai-deb/blob/master/patches/rtai-config-multi-kernel.patch

ShabbyX commented 10 years ago

Great work! I'm reading the script and here are some comments:

I'm guessing the line:

for dir in ${PATH//:/ }; do

will end up having a problem if someone in PATH has whitespace. I don't remember the syntax exactly now, but something along the line of IFS=':' and then just using $PATH or something like that would be the proper way. And of course, quotation is necessary.

There are actually a couple places that quotation is necessary, but once you have a not-so-much-WIP, then I can also add this kind of stuff to it myself, and then test it.

ShabbyX commented 9 years ago

Hi John (@zultron),

I'm trying your debian builds these days (I installed a wheezy in virtual box, so I'd avoid doing it in Ubuntu, but I'll give that a shot later too).

Here are a couple of things I've observed so far, although I played with it very little:

  1. The --config option:

    $ rtai-config -v $(uname -r) --config

    gives

    /rtai/rati-config-UNOFFICIAL

    what's that? Is there a missing prefix? I couldn't find a file with that name.

  2. It would be nice to have a set of additional things installed with the librtai-dev or librtai1 package (I don't which would be more appropriate). For example a script that loads RTAI modules on startup (to be put in /etc/init.d) or one that gives tab completion to rtai-config. I actually have the former script, although it may need some polishing. The later may not be that necessary though, but it would be nice. I can do that too.

    I'll add them to the RTAI repository here (and add a config to install the init script). So you could perhaps update the packages if you liked the idea.


I'll try to build some of my applications one of these days and give you feedback on the default configuration.

Good job by the way

ShabbyX commented 9 years ago

Trying to install the virtual box modules, on the default 3.2 kernel of wheezy everything works, but with the RTAI kernel (having installed both image and header packages, x86 architecture), the build fails.

Tracing it down, the script that tries to invoke the kernel Makefile for information does the following:

    KERN_VER=`uname -r`
    KERN_DIR="/lib/modules/$KERN_VER/build"
    if [ -d "$KERN_DIR" ]; then
        KERN_REL=`make -sC $KERN_DIR --no-print-directory kernelrelease 2>/dev/null || true`
        if [ -z "$KERN_REL" -o "x$KERN_REL" = "x$KERN_VER" ]; then
            return 0
        fi
    fi
    printf "\nThe headers for the current running kernel were not found. If the following\nmodule compilation fails then this could be the reason.\n"

Which fails with the message in the bottom. I tried to invoke that make line myself:

$ make -sC /lib/modules/$(uname -r)/build --no-print-directory kernelrelease
/usr/src/linux-headers-3.8-1-common-rtai.x86/scripts/Makefile.build:44: /usr/src/linux-headers-3.8-1-common-rtai.x86/scripts/basic/Makefile: No such file or directory

which is strange because calling it for the 3.2 kernel also fails with a similar message, even though the build carries through.

The make.log file (from /var/lib/dkms/vboxguest/4.3.10/build/make.log says:

DKMS make.log for vboxguest-4.3.10 for kernel 3.8-1-rtai.x86-686-pae (i686) Mon Sep  8 23:13:24 CEST 2014
make: Entering directory `/usr/src/linux-headers-3.8-1-rtai.x86-686-pae'
  LD      /var/lib/dkms/vboxguest/4.3.10/build/built-in.o
  LD      /var/lib/dkms/vboxguest/4.3.10/build/vboxguest/built-in.o
make[4]: *** No rule to make target `/var/lib/dkms/vboxguest/4.3.10/build/vboxguest/VBoxGuest-linux.o', needed by `/var/lib/dkms/vboxguest/4.3.10/build/vboxguest/vboxguest.o'.  Stop.
make[3]: *** [/var/lib/dkms/vboxguest/4.3.10/build/vboxguest] Error 2
make[2]: *** [_module_/var/lib/dkms/vboxguest/4.3.10/build] Error 2
make[1]: *** [sub-make] Error 2
make: *** [all] Error 2
make: Leaving directory `/usr/src/linux-headers-3.8-1-rtai.x86-686-pae'

which is also weird because come on, no rule for making a .o file?

Anyway, could be some file is missing from the headers package? Like a version.h or some other file generated after kernel configuration?

zultron commented 9 years ago

Thanks for testing!

The UNOFFICIAL thing is the version reported by the ShabbyX RTAI repo, IIRC. Your rtai-config -v $(uname -r) --config command should simply be picking up another rtai-config-$(uname -r) script; try running bash -x rtai-config -v $(uname -r) --config to see exactly what's going on there. (Sorry I can't test right now.)

Your scripts sound like great additions to the packages. The first sounds useful to a general audience, although Machinekit has its own internal utility for loading both RTAI-hal and application modules. I wish RTAI had a general solution for handling both cases.

I'm sorry about the build errors; this regression was reported a while back, but a system meltdown here in the shop has consumed all my attention for a few weeks now. I'll report back when the offending commits have been reverted and packages are rebuilt.

ShabbyX commented 9 years ago

@zultron ok no hurry!

I'll test your recommendations. I'd also try to see if adding the debian repository to Ubuntu would kill everything I hold dear (in a virtual machine of course), to test if the packages can be installed in Ubuntu.

I just realized last night that there's already a script called rtai-load which handles installing both RTAI modules and user modules. It reads from a runinfo file or something with a custom format. That may be useful to you.

I personally have an init script load rtai modules and I keep them loaded all the time, so another script that just handles that would be useful at least to myself.

zultron commented 9 years ago

The Debian archive was updated with new packages last night that fix a problem that prevented even simple out of tree kmods from being built.

Now I thinking this will not solve the problem you encountered above, which looks different, but it's worth a test.

Point me to the virtualbox modules and I'll try to reproduce your problem.

ShabbyX commented 9 years ago

Thanks I just got the update and the build went ok!

ShabbyX commented 9 years ago

@zultron the multi-kernel rtai-config seems to be working great (although I didn't test it with actual many kernels, but I assume you already did). I just wanted to suggest something before I apply the patch.

Right now, if you don't give the -v option, there's a warning on stderr about it, which is fine for informing the user, but could be unpleasant in a script. Furthermore, the majority (I think we agree) use RTAI with a single kernel so the default action of the script is actual all fine and dandy. I think we could either detect if the output is a terminal and print the warning only in that case, or remove the warning altogether.

The former option means old code updating to newer RTAI don't generate warnings and don't have to be modified. The later option makes it easier for someone typing the command on the terminal (avoids repeating -v $(uname -r)), but prevents the feature from being known by the common folk.

I personally think having multiple RTAI installations means you already know what's going on with RTAI and you probably know about -v, so removing the warning is ok with me, but in the end it's your call! Do you think it would be appropriate to remove the warning? Only when the output is not a terminal or altogether?

zultron commented 9 years ago

Hi ShabbyX,

I didn't test very widely, but IIRC did at least test with a regular system make install of RTAI to ensure it behaved reasonably.

I agree that the warning is irritating, now that I've been using it for a while. :-/ I'd rather not do terminal detection, and would prefer to add e.g. a -q ('quiet') or -a (explicit 'auto-detect') option to shut it up, or like you say just remove it completely.

ShabbyX commented 9 years ago

@zultron Ok I think a good compromise which would probably minimize the user error without bugging the average user would be something like this:

If -v is not present, look for rtai-config-* like you do now. If there is none: bitch about it of course. If there is only one, use it and move on without the need for any other options. If there are more than one, then either complain and fail, or if -c (for current) is given (or whatever else specifically for this) then choose the config for the current kernel.

So if you have only a single kernel with rtai, you do rtai-config --option like always. If you have multiple and you wish to work with the current kernel, you do rtai-config -c --option which is not much longer. In a script, you do rtai_config=rtai-config -v VERSION and use $rtai_config.

If you agree, I can change the script accordingly myself. If you think it's too complicated, we can also simplify it by for example always selecting current kernel when there are multiple choices and no -v.

zultron commented 9 years ago

I guess 'current kernel' means 'currently running kernel'? I slightly prefer -r so it's completely clear to build against headers for the running 2.6.32, and not the newer 3.8.13 headers. ;)

So yeah, that sounds great. In my use case, rtai-config is run from autoconf on a builder not running an RTAI kernel, but which should only have a single linux-headers-*-rtai package installed. Auto-detection definitely simplifies the scripts, but the warning is really annoying, and mainly it just needs to fail out if there's any confusion about what to build for. Perfect.

Thanks for taking this on. If you need anything, please ask.

ShabbyX commented 9 years ago

Yes, that's exactly what I meant. Good then, I'll be on it.

ShabbyX commented 9 years ago

@zultron

With some delay, I got to adjust the script today! Give it a spin whenever you can to make sure it works alright and as you would expect it should.

zultron commented 9 years ago

Great, Shabby!

I can't test for a day or two. It's a little hard to see exactly what the changes are until I can pull the patch and look with git diff -w, but the logic looks perfect. I'll build some new kernel packages and point you to them (now building natively on Trusty, in case that's the Ubuntu you're on).

ShabbyX commented 9 years ago

Awesome!

To see the diff, make sure to use --ignore-all-space; the file used four-spaces as indentation, but then tab for two indentations. I just changed it all to spaces for uniformity. Ignoring the whitespace changes, the commit is quite small actually.

ShabbyX commented 9 years ago

I just got word from github that you can actually do that online too, by adding ?w=1 to the address. So you can see the changes here:

https://github.com/ShabbyX/RTAI/commit/61a81774cb122afbe0a83420b0e5921185fc22cf?diff=split&w=1

(github secrets: https://github.com/blog/967-github-secrets)

mrudulachougule commented 8 years ago

Hi, I am working on rtai-4.1 version and i am able to compile rtai debian and installed on Ubuntu 14.04. Now I have below queries on rtai_lxrt module: 1) why rtai_lxrt.ko is soft link of rtai_sched.ko? 2) In rtai-3.8 version rtai_lxrt is not linked to any other module any specific reason? 3) I need rtai_lxrt individual module to be imported any idea how to do this? 4) I have python back end code which imports few of the module from rtai, since rtai_lxrt is not inserted so i'm not able to import the same? 5) can I include rtai_lxrt header file to resolve python and rtai binding issue? 6) CONFIG_RTAI_LXRT_USE_LINUX_SYSCALL option is not available to configure in rtai-4.1 but same is available, any idea how to enable / explicitly overwrite this option? 7) How to get rtai_lxrt.ko as application scheduler, without soft link of rtai_sched?

Please share your views to resolve these issues , I'm stuck!!! Appreciate your suggestions!!

Thanks

ShabbyX commented 8 years ago

@mrudulachougule Please open a new issue with unrelated questions. I opened and answered an issue for you here.