Unable to install GG onto arm device.

[matthewtanner]

I am prototyping GG core devices on armv7l I'm using the easy installer before adding the components to our filesystem and implementing fleet provisioning. I have things working fine on a Ubuntu amd64 VM using the same steps. It appears provisioning succeeds and everything is extracted and copied to GG root (/greengrass/v2), but nucleus fails when it runs.

It appears to be failing to link libaws-crt-jni.so

output

sudo -E java -Droot="/greengrass/v2" -Dlog.store=FILE -jar ./GreengrassInstaller/lib/Greengrass.jar --aws-region us-east-1 --thing-name <redacted> --thing-group-name <redacted> --component
-default-user ggc_user:ggc_group --provision true --setup-system-service false
Provisioning AWS IoT resources for the device with IoT Thing Name: [<redacted>]...
Found IoT policy "GreengrassV2IoTThingPolicy", reusing it
Creating keys and certificate...
Attaching policy to certificate...
Creating IoT Thing "<redacted>"...
Attaching certificate to IoT thing...
Successfully provisioned AWS IoT resources for the device with IoT Thing Name: [<redacted>]!
Adding IoT Thing [<redacted>] into Thing Group: [<redacted>]...
IoT Thing Group "<redacted>" already existed, reusing it
Successfully added Thing into Thing Group: [<redacted>]
Setting up resources for aws.greengrass.TokenExchangeService ... 
Attaching TES role policy to IoT thing...
No managed IAM policy found, looking for user defined policy...
IAM policy named "GreengrassV2TokenExchangeRoleAccess" already exists. Please attach it to the IAM role if not already
Configuring Nucleus with provisioned resource details...
Downloading Root CA from "https://www.amazontrust.com/repository/AmazonRootCA1.pem"
Created device configuration
Successfully configured Nucleus with provisioned resource details!
Launching Nucleus...
Error while trying to setup Greengrass Nucleus
java.lang.UnsatisfiedLinkError: /tmp/AWSCRT_16163488345316011430libaws-crt-jni.so: /tmp/AWSCRT_16163488345316011430libaws-crt-jni.so: internal error
    at java.base/java.lang.ClassLoader$NativeLibrary.load0(Native Method)
    at java.base/java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2450)
    at java.base/java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2506)
    at java.base/java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2705)
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2635)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1854)
    at software.amazon.awssdk.crt.CRT.extractAndLoadLibrary(CRT.java:337)
    at software.amazon.awssdk.crt.CRT.loadLibraryFromJar(CRT.java:371)
    at software.amazon.awssdk.crt.CRT.<clinit>(CRT.java:45)
    at software.amazon.awssdk.crt.CrtResource.<clinit>(CrtResource.java:104)
    at com.aws.greengrass.ipc.IPCEventStreamService.startup(IPCEventStreamService.java:97)
    at com.aws.greengrass.lifecyclemanager.KernelLifecycle.launch(KernelLifecycle.java:152)
    at com.aws.greengrass.lifecyclemanager.Kernel.launch(Kernel.java:274)
    at com.aws.greengrass.easysetup.GreengrassSetup.performSetup(GreengrassSetup.java:347)
    at com.aws.greengrass.easysetup.GreengrassSetup.main(GreengrassSetup.java:274)

I have a an AWS re:Post question active on issue 1 here ,however I think I need deeper technical assistance here.

I am running the installer on linux armv7l glib architecture. it is a custom Linux distribution using busybox/ash shell. Nucleus version v2.12.6 openJDK 11.0.22

[/]# uname -m
armv7l
[/]# uname -o
GNU/Linux

Xilinx Zynq-7000 SoC

[/]# cat /proc/cpuinfo 
processor   : 0
model name  : ARMv7 Processor rev 0 (v7l)
BogoMIPS    : 666.66
Features    : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x3
CPU part    : 0xc09
CPU revision    : 0

processor   : 1
model name  : ARMv7 Processor rev 0 (v7l)
BogoMIPS    : 666.66
Features    : half thumb fastmult vfp edsp neon vfpv3 tls vfpd32 
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x3
CPU part    : 0xc09
CPU revision    : 0

Hardware    : Xilinx Zynq Platform
Revision    : 0003
Serial      : 0000000000000000

[jrcarbAWS]

aws-crt dependency was updated after the 2.12.4 release. Can you confirm what Nucleus version has the issue on arm? You mention 2.14.x and in the re:post it seems like the opposite and you were using 2.12.4.

I tested on my own version of armv7l GNU/Linux with no issues, so this may only be an issue with your custom distribution that you mentioned. I am not sure if replacing the file between version, will help (you can try). Are you able to downgrade your version, to unblock yourself?

I also downloaded and unzipped both v2.12.4 and v2.12.6 jars, and both /linux/armv7/glibc/libaws-crt-jni.so are of "hard-float", can you confirm what version you saw soft-float?

[matthewtanner]

Ok, I found out that the version 2.12.4 of GG that I was using had a version of /linux/armv7/glibc/libaws-crt-jni.so that a developer had cross compiled to soft float. That solves the difference in my versions. It looks like CRT has known limitations around auto selecting armv7 musl vs glibc, so I'm guessing it also doesn't know what to do when the host processor can't support hard float.

We will include the soft float version in our filesystem matching whatever nucleus is providing.

Thanks and sorry for the confusion.

[jrcarbAWS]

Glad you were able to figure out what was going on, and unblock yourself. Will close since it doesn't seem like there is an issue.