jetsonhacks / buildJetsonXavierKernel

MIT License
26 stars 11 forks source link

New Kernel install on Jetson AGX #9

Open fmolinelli opened 3 years ago

fmolinelli commented 3 years ago

Hi, thank you so much for your great work.

Currently I'm trying to enable open-iscsi on NVIDIA Jetson Xavier AGX module, in order to use on it rancher/longhorn distributed storage.

The standard kernel isn't built with open-iscsi support, so I rebuilt it using the following kernel options: CONFIG_CRYPTO=y CONFIG_CRYPTO_CRC32C=y CONFIG_ISCSI_TCP=m CONFIG_SCSI_ISCSI_ATTRS=m CONFIG_SCSI_LOWLEVEL=y CONFIG_BLK_DEV=y CONFIG_INET=y

I thoroughly followed your README and didn't see anything wrong

Kernel and modules build correctly, the board reboots fine with the new kernel, at least it seems so (see later...) but 'modprobe iscsi_tcp' fails: blueadmin@BL-AGX01:/wdir/kernel/buildJetsonXavierKernel$ sudo modprobe iscsi_tcp [sudo] password for blueadmin: modprobe: ERROR: could not insert 'iscsi_tcp': Unknown symbol in module, or unknown parameter (see dmesg): blueadmin@BL-AGX01:/wdir/kernel/buildJetsonXavierKernel$ dmesg | tail [ 4021.062113] scsi_transport_iscsi: Unknown symbol bsg_request_fn (err 0) [ 4423.947586] scsi_transport_iscsi: Unknown symbol bsg_setup_queue (err 0) [ 4423.947789] scsi_transport_iscsi: Unknown symbol bsg_job_done (err 0) bsglib should be included... blueadmin@BL-AGX01:~/wdir/kernel/buildJetsonXavierKernel$ grep BSG /usr/src/kernel/kernel-4.9/.config CONFIG_BLK_DEV_BSG=y CONFIG_BLK_DEV_BSGLIB=y

Here's my Xavier AGX version: blueadmin@BL-AGX01:~$ ./jetsonInfo.py NVIDIA Jetson AGX Xavier [16GB] L4T 32.6.1 [ JetPack 4.6 ] Ubuntu 18.04.6 LTS Kernel Version: 4.9.253-tegra CUDA 10.2.300 CUDA Architecture: 7.2 OpenCV version: 4.1.1 OpenCV Cuda: NO CUDNN: 8.2.1.32 TensorRT: 8.0.1.6 Vision Works: 1.6.0.501 VPI: ii libnvvpi1 1.1.12 arm64 NVIDIA Vision Programming Interface library Vulcan: 1.2.70

Here are the available scsi related kernel modules: blueadmin@BL-AGX01:~$ find /lib/modules/$(uname -r) -type f -name scs /lib/modules/4.9.253-tegra/kernel/drivers/scsi/iscsi_tcp.ko /lib/modules/4.9.253-tegra/kernel/drivers/scsi/libiscsi_tcp.ko /lib/modules/4.9.253-tegra/kernel/drivers/scsi/scsi_transport_iscsi.ko /lib/modules/4.9.253-tegra/kernel/drivers/scsi/scsi_transport_srp.ko /lib/modules/4.9.253-tegra/kernel/drivers/scsi/libiscsi.ko

and here's the iscsi_tcp modinfo: blueadmin@BL-AGX01:~$ modinfo iscsi_tcp filename: /lib/modules/4.9.253-tegra/kernel/drivers/scsi/iscsi_tcp.ko license: GPL description: iSCSI/TCP data-path author: Mike Christie michaelc@cs.wisc.edu, Dmitry Yusupov dmitry_yus@yahoo.com, Alex Aizman itn780@yahoo.com depends: libiscsi,libiscsi_tcp,scsi_transport_iscsi intree: Y vermagic: 4.9.253-tegra SMP preempt mod_unload modversions aarch64 parm: max_lun:uint parm: debug_iscsi_tcp:Turn on debugging for iscsi_tcp module Set to 1 to turn on, and zero to turn off. Default is off. (int)

Thinking I missed some kernel options... or that I failed to install it on the boot dir:

When testing a custom kernel, it is recommended that you create a backup of

the original kernel and add a new entry to this file so that the device can

fallback to the original kernel. To do this:

#

1, Make a backup of the original kernel

sudo cp /boot/Image /boot/Image.backup

#

2, Copy your custom kernel into /boot/Image

#

3, Uncomment below menu setting lines for the original kernel

#

4, Reboot

LABEL backup MENU LABEL backup kernel LINUX /boot/Image.backup INITRD /boot/initrd APPEND ${cbootargs}

my new kernel's compil date should be semothing like MON Oct 11 12:xx:xx .....

Pls can you help me getting the new kernel booted in place of the older one?

Best regards Flavio

PS as per NVIDIA Kernel Customization Guide, the boot loader should load the given Image even if not signed ....

jetsonhacks commented 3 years ago

Are you running from the eMMC or a NVMe SSD?

fmolinelli commented 3 years ago

Hi.

booting from eMMC, have a NVMe only for data.

Thank you! Flavio

Il 12/10/2021 03:05, Jetsonhacks ha scritto:

Are you running from the eMMC or a NVMe SSD?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jetsonhacks/buildJetsonXavierKernel/issues/9#issuecomment-940570417, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCOLIQFJW7CLVRGFAUDJ6LUGOCV7ANCNFSM5FYQO6VQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Flavio Molinelli

Networking & Security Consulting s.r.l. -www.nesecon.com- Tel. ++39.011.9904163

ATTENZIONE!! Il presente messaggio ha contenuto confidenziale, e la sua lettura, allegati compresi, e' riservata esclusivamente ai destinatari previsti. Nel caso riteniate di non essere uno dei destinatari previsti, siete pregati di distruggere il messaggio e di informarne il mittente. Il trattameno dei dati personali è gestito da NSC s.r.l. in conformità con la normativa vigente GDPR Regulation (EU) 2016/679 - Dlgs. 101/2018

WARNING!! This message contains confidential information, and it is intended to be read, attachments included, only by the intended recipients. If you believe not to be one of the intended recipients, please destroy the message and inform the sender. Personal data are processed by NSC s.r.l. as per GDPR Regulation (EU) 2016/679 - Dlgs. 101/2018

jetsonhacks commented 3 years ago

What does the rest of your extlinux.conf look like? I would suggest that you take a look at the serial console and see where it is trying to load the Image. Also, check to make sure that the modules are in the correct place.

fmolinelli commented 3 years ago

hi,

/What does the rest of your extlinux.conf look like?/

Pls see listings below

/I would suggest that you take a look at the serial console and see where it is trying to load the Image./

AFAIK the console should be the 4th usbserial device on the left microUSB plug, and should be alsways on, even when the AGX is off (but powered on... ) but I spent two days getting at it wuthout any success:   - When I connect my ubuntu ( or macOS) desktop to the left micro-USB plug I don't see any device coming up in dmesg... so no way to monitor the console output from there.   - I only get access to the console at the right USB-C (multifunction Flash/debug) as /dev/ttyACM0:115200:8N1, but this seems to be a serial TTY terminal, not a system console, and it is reset at every boot, so I cannot trace the pre-image boot operations. QUESTION: is there an early boot stpe's  log written somewhere else?

/Also, check to make sure that the modules are in the correct place. /the modules seems to be in the right place, as they are loaded at boot, generating the error messages seen in dmesg boot sequence.

Also asked support from open-iscsi' maintainer Chris Leech, which told that 'The missing symbols should be enabled with CONFIG_BLK_DEV_BSGLIB'

In fact the booting original kernel Image has that option unset, while my compiled kernel does have it set. That could be the reason the modules aren't working with the original NVIDIA Image

QUESTION: is there another way to boot the compiled Image, like placing it on a USB drive, or microSD card? it would be fine in case of compil errors / mistakes in options settings / etc: simply plug out the device and you'd have your AGX back running with old kernel... ////

//

here's  y /boot/extlinux/extlinux.conf:

@.***:~$ cat /boot/extlinux/extlinux.conf TIMEOUT 30 DEFAULT primary

MENU TITLE L4T boot options

LABEL primary       MENU LABEL primary kernel       LINUX /boot/Image       INITRD /boot/initrd       APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4 console=ttyTCU0,115200n8 console=tty0 fbcon=map:0 net.ifnames=0 rootfstype=ext4

When testing a custom kernel, it is recommended that you create a

backup of

the original kernel and add a new entry to this file so that the

device can

fallback to the original kernel. To do this:

#

1, Make a backup of the original kernel

      sudo cp /boot/Image /boot/Image.backup

#

2, Copy your custom kernel into /boot/Image

#

3, Uncomment below menu setting lines for the original kernel

#

4, Reboot

LABEL backup     MENU LABEL backup kernel     LINUX /boot/Image.backup     INITRD /boot/initrd     APPEND ${cbootargs}


the  file extlinux.conf.nv-update-extlinux-backup differences:

@.:/boot/extlinux$ ls extlinux.conf  extlinux.conf.nv-update-extlinux-backup @.:/boot/extlinux$ diff * 25,29c25,29 < LABEL backup <     MENU LABEL backup kernel <     LINUX /boot/Image.backup <     INITRD /boot/initrd <     APPEND ${cbootargs}

LABEL backup

    MENU LABEL backup kernel

    LINUX /boot/Image.backup

    INITRD /boot/initrd

    APPEND ${cbootargs}


and here's my /boot tree


@.***:~$ sudo tree /boot /boot ├── dtb │   ├── kernel_tegra194-p2888-0001-p2822-0000.dtb │   └── kernel_tegra194-p2888-0001-p2822-0000.dtb.sig ├── extlinux │   ├── extlinux.conf │   └── extlinux.conf.nv-update-extlinux-backup ├── grub │   └── grubenv ├── Image ├── Image.backup ├── Image.sig ├── Image.t19x ├── Image.t19x.sig ├── initrd ├── initrd.img -> initrd.img-4.9.201-tegra ├── initrd.img-4.9.201-tegra ├── initrd.sig ├── initrd.t19x ├── initrd.t19x.sig ├── tegra186-p3636-0001-p3509-0000-a01-adafruit-sph0645lm4h.dtbo ├── tegra186-p3636-0001-p3509-0000-a01-adafruit-uda1334a.dtbo ├── tegra186-p3636-0001-p3509-0000-a01.dtb ├── tegra186-p3636-0001-p3509-0000-a01-fe-pi-audio.dtbo ├── tegra186-p3636-0001-p3509-0000-a01-hdr40.dtbo ├── tegra186-p3636-0001-p3509-0000-a01-m2ke.dtbo ├── tegra186-p3636-0001-p3509-0000-a01-respeaker-4-mic-array.dtbo ├── tegra186-p3636-0001-p3509-0000-a01-respeaker-4-mic-lin-array.dtbo ├── tegra186-quill-p3310-1000-a00-00-base.dtb ├── tegra186-quill-p3310-1000-as-0888.dtb ├── tegra186-quill-p3310-1000-c03-00-base-adafruit-sph0645lm4h.dtbo ├── tegra186-quill-p3310-1000-c03-00-base-adafruit-uda1334a.dtbo ├── tegra186-quill-p3310-1000-c03-00-base-csi.dtbo ├── tegra186-quill-p3310-1000-c03-00-base.dtb ├── tegra186-quill-p3310-1000-c03-00-base-fe-pi-audio.dtbo ├── tegra186-quill-p3310-1000-c03-00-base-hdr30.dtbo ├── tegra186-quill-p3310-1000-c03-00-base-hdr40.dtbo ├── tegra186-quill-p3310-1000-c03-00-base-m2ke.dtbo ├── tegra186-quill-p3310-1000-c03-00-base-respeaker-4-mic-array.dtbo ├── tegra186-quill-p3310-1000-c03-00-base-respeaker-4-mic-lin-array.dtbo ├── tegra186-quill-p3310-1000-c03-00-dsi-hdmi-dp.dtb ├── tegra186-quill-p3489-0888-a00-00-base.dtb ├── tegra186-quill-p3489-1000-a00-00-ucm1.dtb ├── tegra186-quill-p3489-1000-a00-00-ucm2.dtb ├── tegra194-p2888-0001-p2822-0000-adafruit-sph0645lm4h.dtbo ├── tegra194-p2888-0001-p2822-0000-adafruit-uda1334a.dtbo ├── tegra194-p2888-0001-p2822-0000-csi.dtbo ├── tegra194-p2888-0001-p2822-0000.dtb ├── tegra194-p2888-0001-p2822-0000.dtb.sig ├── tegra194-p2888-0001-p2822-0000-fe-pi-audio.dtbo ├── tegra194-p2888-0001-p2822-0000-hdr40.dtbo ├── tegra194-p2888-0001-p2822-0000-m2ke.dtbo ├── tegra194-p2888-0001-p2822-0000-maxn.dtb ├── tegra194-p2888-0001-p2822-0000-maxn.dtb.sig ├── tegra194-p2888-0001-p2822-0000-respeaker-4-mic-array.dtbo ├── tegra194-p2888-0001-p2822-0000-respeaker-4-mic-lin-array.dtbo ├── tegra194-p2888-0004-e3900-0000-adafruit-sph0645lm4h.dtbo ├── tegra194-p2888-0004-e3900-0000-adafruit-uda1334a.dtbo ├── tegra194-p2888-0004-e3900-0000-csi.dtbo ├── tegra194-p2888-0004-e3900-0000.dtb ├── tegra194-p2888-0004-e3900-0000.dtb.sig ├── tegra194-p2888-0004-e3900-0000-dual-imx274.dtbo ├── tegra194-p2888-0004-e3900-0000-fe-pi-audio.dtbo ├── tegra194-p2888-0004-e3900-0000-hdr40.dtbo ├── tegra194-p2888-0004-e3900-0000-imx274.dtbo ├── tegra194-p2888-0004-e3900-0000-respeaker-4-mic-array.dtbo ├── tegra194-p2888-0004-e3900-0000-respeaker-4-mic-lin-array.dtbo ├── tegra194-p2888-0008-p2822-0000.dtb ├── tegra194-p2888-0008-p2822-0000.dtb.sig ├── tegra194-p2888-as-0006-p2822-0000.dtb ├── tegra194-p2888-as-0006-p2822-0000.dtb.sig ├── tegra194-p2888-as-p3668-p2822-0000.dtb ├── tegra194-p2888-as-p3668-p2822-0000.dtb.sig ├── tegra194-p3668-all-p3509-0000-adafruit-sph0645lm4h.dtbo ├── tegra194-p3668-all-p3509-0000-adafruit-uda1334a.dtbo ├── tegra194-p3668-all-p3509-0000-camera-imx219-dual.dtbo ├── tegra194-p3668-all-p3509-0000-camera-imx477-dual.dtbo ├── tegra194-p3668-all-p3509-0000-camera-imx477-imx219.dtbo ├── tegra194-p3668-all-p3509-0000.dtb ├── tegra194-p3668-all-p3509-0000.dtb.sig ├── tegra194-p3668-all-p3509-0000-fe-pi-audio.dtbo ├── tegra194-p3668-all-p3509-0000-hdr40.dtbo ├── tegra194-p3668-all-p3509-0000-m2ke.dtbo ├── tegra194-p3668-all-p3509-0000-respeaker-4-mic-array.dtbo └── tegra194-p3668-all-p3509-0000-respeaker-4-mic-lin-array.dtbo

3 directories, 81 files


Best Regards Flavio

Il 12/10/2021 19:45, Jetsonhacks ha scritto:

What does the rest of your extlinux.conf look like? I would suggest that you take a look at the serial console and see where it is trying to load the Image. Also, check to make sure that the modules are in the correct place.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jetsonhacks/buildJetsonXavierKernel/issues/9#issuecomment-941232065, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCOLIWTI4Y5CWM45B5I763UGRX2VANCNFSM5FYQO6VQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Flavio Molinelli

Networking & Security Consulting s.r.l. -www.nesecon.com- Tel. ++39.011.9904163

ATTENZIONE!! Il presente messaggio ha contenuto confidenziale, e la sua lettura, allegati compresi, e' riservata esclusivamente ai destinatari previsti. Nel caso riteniate di non essere uno dei destinatari previsti, siete pregati di distruggere il messaggio e di informarne il mittente. Il trattameno dei dati personali è gestito da NSC s.r.l. in conformità con la normativa vigente GDPR Regulation (EU) 2016/679 - Dlgs. 101/2018

WARNING!! This message contains confidential information, and it is intended to be read, attachments included, only by the intended recipients. If you believe not to be one of the intended recipients, please destroy the message and inform the sender. Personal data are processed by NSC s.r.l. as per GDPR Regulation (EU) 2016/679 - Dlgs. 101/2018

fmolinelli commented 3 years ago

HI,

nope getting it working on Xavier AGX ... but...

it worked on Xavier NX !!!!!

I did exacly the same procedure on NX as I did on AGX, and voilà, NX booted my newly compiled kernel and open-iscsi was up and running:

$ uname -a Linux JXNX01 4.9.253-tegra #1 SMP PREEMPT Thu Oct 14 12:09:37 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux $ systemctl status iscsid ● iscsid.service - iSCSI initiator daemon (iscsid)    Loaded: loaded (/lib/systemd/system/iscsid.service; enabled; vendor preset: enabled)    Active: active (running) since Thu 2021-10-14 14:34:10 CEST; 3min 16s ago      Docs: man:iscsid(8)  Main PID: 5651 (iscsid)     Tasks: 2 (limit: 4915)    CGroup: /system.slice/iscsid.service            ├─5647 /sbin/iscsid            └─5651 /sbin/iscsid

ott 14 14:34:09 JXNX01 systemd[1]: Starting iSCSI initiator daemon (iscsid)... ott 14 14:34:10 JXNX01 iscsid[5562]: iSCSI logger with pid=5647 started! ott 14 14:34:10 JXNX01 systemd[1]: Started iSCSI initiator daemon (iscsid). ott 14 14:34:11 JXNX01 iscsid[5647]: iSCSI daemon with pid=5651 started!

Now it seems to be an AGX booting related issue... But accordingly to NVIDIA jetson Developer Guide AGX & NX share the same booting flow....

do you have any idea about it?

(the difference between two boards is that AGX is booting from internal eMMC, while NX is booting from microSD ,,, would that matter?)

Hoping to get some hint from you!

Best Regards Flavio

PS Going ondoing the same  with a Jetson Nano 2G: I'll report my results!

Il 12/10/2021 19:45, Jetsonhacks ha scritto:

What does the rest of your extlinux.conf look like? I would suggest that you take a look at the serial console and see where it is trying to load the Image. Also, check to make sure that the modules are in the correct place.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jetsonhacks/buildJetsonXavierKernel/issues/9#issuecomment-941232065, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCOLIWTI4Y5CWM45B5I763UGRX2VANCNFSM5FYQO6VQ. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

-- Flavio Molinelli

Networking & Security Consulting s.r.l. -www.nesecon.com- Tel. ++39.011.9904163

ATTENZIONE!! Il presente messaggio ha contenuto confidenziale, e la sua lettura, allegati compresi, e' riservata esclusivamente ai destinatari previsti. Nel caso riteniate di non essere uno dei destinatari previsti, siete pregati di distruggere il messaggio e di informarne il mittente. Il trattameno dei dati personali è gestito da NSC s.r.l. in conformità con la normativa vigente GDPR Regulation (EU) 2016/679 - Dlgs. 101/2018

WARNING!! This message contains confidential information, and it is intended to be read, attachments included, only by the intended recipients. If you believe not to be one of the intended recipients, please destroy the message and inform the sender. Personal data are processed by NSC s.r.l. as per GDPR Regulation (EU) 2016/679 - Dlgs. 101/2018

fmolinelli commented 3 years ago

Hi

eventually I found a way to solve this issue: it could be a workaround, it could be a feature.... at least for develpment phase.

That's my last story:

2) It never look at eMMC partition 1 (APP, where /boot resides)

3) When going to SD, USB, NVME, it looks for an active partition with EXT2/4 and /boot and /boot/extlinux content...

@.:~$ uname -a Linux BL-AGX01 4.9.253-tegra #1 SMP PREEMPT Mon Oct 11 12:35:16 CEST 2021 aarch64 aarch64 aarch64 GNU/Linux @.:~$ systemctl status iscsid ● iscsid.service - iSCSI initiator daemon (iscsid)    Loaded: loaded (/lib/systemd/system/iscsid.service; enabled; vendor preset: enabled)    Active: active (running) since Thu 2021-10-14 18:52:56 CEST; 3min 14s ago      Docs: man:iscsid(8)  Main PID: 6195 (iscsid)     Tasks: 2 (limit: 4915)    CGroup: /system.slice/iscsid.service            ├─6194 /sbin/iscsid            └─6195 /sbin/iscsid

ott 14 18:52:56 BL-AGX01 systemd[1]: Starting iSCSI initiator daemon (iscsid)... ott 14 18:52:56 BL-AGX01 iscsid[6169]: iSCSI logger with pid=6194 started! ott 14 18:52:56 BL-AGX01 systemd[1]: Started iSCSI initiator daemon (iscsid). ott 14 18:52:57 BL-AGX01 iscsid[6194]: iSCSI daemon with pid=6195 started! @.***:~$

Conclusion:

 - AGX board no more boot from mmcblk0p1/boot, but only from signed images residing into 'kernel' or 'kernel_b' partitions (p29 and p30, could be)  - AGX booting partitions can only be flashed from SDK host (if the imgae building tree is OK),  to AGX in recovery mode  - /boot/Image can be retrieved from microSD, USB, NVME devices, along with extlinux config: no signing is required, if 'security fuse' hasn't been burned yet: this  allows for remote update, in place of OTA update, and for testing new kernels without burning them on flash (put a USB drive with /boot dir on it and go testing: if it doesn't work... remove and you have your AGX back working with old kernel...

Just a thought: AGX L4T online update do work completely in place: no SDK host is required... so there's a way to burn the 'kernel' partition with the 'right' image... there's only a need to understand how to do that... here's some reference could help finding it:  - https://forums.developer.nvidia.com/t/flashing-a-new-kernel-natively-on-xavier/67559/6 (where are the scripts...?)  - https://forums.developer.nvidia.com/t/custom-kernel-without-uboot-solved/65075 (why eMMC /boot is used no more...)

Hoping this would help other developers

Best regards Flavio

--

Flavio Molinelli

Networking & Security Consulting s.r.l. -www.nesecon.com- Tel. ++39.011.9904163

ATTENZIONE!! Il presente messaggio ha contenuto confidenziale, e la sua lettura, allegati compresi, e' riservata esclusivamente ai destinatari previsti. Nel caso riteniate di non essere uno dei destinatari previsti, siete pregati di distruggere il messaggio e di informarne il mittente. Il trattameno dei dati personali è gestito da NSC s.r.l. in conformità con la normativa vigente GDPR Regulation (EU) 2016/679 - Dlgs. 101/2018

WARNING!! This message contains confidential information, and it is intended to be read, attachments included, only by the intended recipients. If you believe not to be one of the intended recipients, please destroy the message and inform the sender. Personal data are processed by NSC s.r.l. as per GDPR Regulation (EU) 2016/679 - Dlgs. 101/2018

jetsonhacks commented 3 years ago

@fmolinelli I am glad you got something to work.

Without the serial debug kernel, it would have been very difficult indeed to figure out what was going on. I'll leave your issue open so that others may benefit.