fenrus75
commented
3 years ago interesting .. which release number are you using for this? we're currently changing a bit how avx2/avx512 support is done, and it could be that you lock that in at image create time (say on an AVX512 capable machine) but then run it on an AVX2-only machine ?
On Fri, Oct 15, 2021 at 9:26 AM Michael Sverdlik @.***> wrote:
I'm chasing a weird issue in which Clear Linux based AMIs that I create end up with segfaults during instance startup. These segfaults prevent services such ucd from running. Can't provide core dump either as it looks like systemd-coredump segfults as well.
Reproducing
- Start with ami-07d9f0e2130e032fe (all of this done in ap-southeast-2 region) and launch t3 family instance (I used t3.medium).
- Once the instance is up, create an image from it.
- Launch a new t2 family instance (I used t2.medium) from the new AMI you created in the previous step.
- Connect and inspect the journal:
kernel: ucd[159]: segfault at fffffffffffffffd ip 00007f9ee6334424 sp 00007fff2a191600 error 5 in libglib-2.0.so.0.7000.0[7f9ee62b3000+e1000] kernel: Code: 48 89 44 24 28 48 89 05 d2 da 0e 00 48 3d ff 01 00 00 0f 86 96 04 00 00 c4 e2 f8 f3 4c 24 28 48 89 44 24 30 0f 85 9c 04 00 00 <62> 61 fd 28 6f 3d 52 d0 0e 00 48 8d 3d 96 77 07 00 62 61 fe 28 7f
systemd[1]: ucd.service: Main process exited, code=dumped, status=11/SEGV systemd[1]: ucd.service: Failed with result 'core-dump'. systemd[1]: Failed to start micro-config-drive job.
kernel: systemd-coredum[174]: segfault at 1 ip 00007f7b2c21e825 sp 00007ffc98715820 error 4 in libzstd.so.1.5.0[7f7b2c213000+108000]
kernel: Code: 6d 28 77 71 62 01 05 00 ef ff 62 61 7f 28 7f 7d 01 62 61 7f 28 7f 78 01 48 85 f6 74 0d 4d 85 e4 75 7a 48 89 f7 e8 0b 48 ff ff <62> 01 05 00 ef ff 62 61 7f 28 7f bb a8 0d 00 00 62 61 fd 08 7e bb
If you try to execute ucd directly:
@.***~ $ /usr/bin/ucd Segmentation fault (core dumped)
This doesn't seem to happen in all of the instances. I managed to run the m5 family instance from the image without issues but it seems to happen again if trying to start t3a family one:
kernel: traps: ucd[153] trap invalid opcode ip:7fee43bb7424 sp:7ffde964d740 error:0 in libglib-2.0.so.0.7000.0[7fee43b36000+e1000]
systemd[1]: ucd.service: Main process exited, code=dumped, status=4/ILL systemd[1]: ucd.service: Failed with result 'core-dump'. systemd[1]: Failed to start micro-config-drive job.
Workaround If I create AMI from a t2 family, it seems to start without issues on the t3 or t3a families without a problem.
Is this use case even supported? I didn't found much info online about creating new AMIs based on Clear Linux. And UCD docs don't explain how to "prepare" an instance before making AMI.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/clearlinux/distribution/issues/2449, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ54FMH546L7GECEGNGFUTUHBI47ANCNFSM5GCKNKYQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
m1keil
thiagomacieira
I'm chasing a weird issue in which Clear Linux based AMIs that I create end up with segfaults during instance startup. These segfaults prevent services such ucd from running. Can't provide core dump either as it looks like systemd-coredump segfults as well.
Reproducing
If you try to execute ucd directly:
This doesn't seem to happen in all of the instances. I managed to run the m5 family instance from the image without issues but it seems to happen again if trying to start t3a family one:
Workaround If I create AMI from a t2 family, it seems to start without issues on the t3 or t3a families.
Is this use case even supported? I didn't find much info online about creating new AMIs based on Clear Linux. And UCD docs don't explain how to "prepare" an instance before making a new AMI (like the need to remove
/var/lib/cloud/aws-user-data)