segfault when running sesictrl in aaronsmithtv/hbuild:20.5.278-base

[sabjorn]

problem

When trying to run sesictrl in aaronsmithtv/hbuild:20.5.278-base the application segfaults. This makes it impossible to obtain an Houdini license within the container.

steps to reproduce

start hserver

hserver -Q --render-only -S https://www.sidefx.com/license/sesinetd --clientid <client id> --clientsecret <client secret>

run sesictrl

sesictrl login --email <username/email> --password <password>

error log

after starting hserver

sh: 1: systemctl: not found
Unable to connect to hserver. Attempting to restart hserver...
sh: 1: systemctl: not found
Successfully started hserver after 4995500ms.
Successfully changed server listing.

after running sesictrl

You are not logged into the license server. [Error L01]
23: Fatal error: Segmentation fault (sent by pid -24)
-- TRACEBACK BEGIN --
Traceback from sesictrl 20.5.278 (Compiled on linux-x86_64-gcc9.3):
malloc_usable_size <sesictrl>
malloc_usable_size <sesictrl>
malloc_usable_size <sesictrl>
malloc_usable_size <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
__libc_start_main <libc.so.6>
mremap <sesictrl>
-- TRACEBACK END --

additional information

I tried installing systemctl to see if that would make hserver function correctly, but instead got:

ERROR:systemctl:Unit hserver.service could not be found.
Server shutdown started.

[aaronsmithtv]

Hey @sabjorn, thanks for sharing your issue! I think this is likely down to the list of libraries required to run Houdini being extended with 20.5; This is definitely something that will get looked at in the coming weeks :)

[eloop]

I notice H20.5 has converted over to using systemd units instead of installing simple init scripts for running up daemons. So possibly related to this issue - https://stackoverflow.com/questions/54079586/make-systemctl-work-from-inside-a-container-in-a-debian-stretch-image

[sabjorn]

Hey @sabjorn, thanks for sharing your issue! I think this is likely down to the list of libraries required to run Houdini being extended with 20.5; This is definitely something that will get looked at in the coming weeks :)

yeah -- this is the type of segfault I would expect when the libgcc shared library used when compiling the application does not match the one running on the system. However -- I simply can't find any documentation from Houdini on what versions of things are being used.

[eloop]

Also this is mentioned in the SideFX changelogs for their Dockerfile example, so probbaly worth copying from them.

20.5.300 - Updated the Ubuntu Dockerfile to use Ubuntu 22.04 and included more required packages.

https://www.sidefx.com/download/daily-builds/?production=true&python3=true&docker=true

A quick look at them would suggest that they haven't been tested much, but maybe the required libs have been updated properly.

[eloop]

Also note, you'll need the systemctl stuff installed before running the houdini install.

[sabjorn]

Also this is mentioned in the SideFX changelogs for their Dockerfile example, so probbaly worth copying from them.

20.5.300 - Updated the Ubuntu Dockerfile to use Ubuntu 22.04 and included more required packages.

https://www.sidefx.com/download/daily-builds/?production=true&python3=true&docker=true

A quick look at them would suggest that they haven't been tested much, but maybe the required libs have been updated properly.

These dockerfiles are included with the installer and I checked them previously -- however I did modify to user debian-slim so it could be the version of Ubuntu they are referencing has the correct configuration.

But it could definitely be that they haven't been testing these and so it's broken and they don't know.

[aaronsmithtv]

Thanks for the assistance @eloop, I missed that changelog relating to systemd. Looks like it goes back to 20.0 - if that's the case then any image in the last 9 months may have similar issues! Hopefully the installer distinguishes installations by whether or not systemd is present on the image. Though this may be why SideFX opts to use Ubuntu 22.04.

I'll add looking at this to the investigation I'd like to make regarding installation libraries.

[aaronsmithtv]

@sabjorn After a day of looking at this, I am pretty sure there's some kind of issue with sesictrl login. Tried it on both a Linux machine with systemd, and in a variety of different Docker containers, including those with Ubuntu, Debian, and loads of various libaries (including those included by SideFX in their Dockerfile). Segfaults every time.

I've sent an email to support - let's see where it goes :)

[sabjorn]

@aaronsmithtv thanks for putting this work in!

[aaronsmithtv]

Sorry to message so quickly after @sabjorn but I think I've cracked it with the help of @captainhammy!

I think the args passed into hserver have changed since the last update. It's really weird:

Run this command, which you'll notice now uses the explicit flag --host to denote the license server path.

hserver --clientid "<your id>" --clientsecret "<your secret>" --host "https://www.sidefx.com/license/sesinetd"

This also now works for me without supplying credentials, which seems like the smart thing to be honest given the API login.

sesictrl login

This now works for me too! The OpenCL exception will get removed in a future update, as I'm already planning to add a basic CPU locked installation for the library anyway.

root@8f1911f15ec4:/# hython
OpenCL Exception: No OpenCL platform found. (-32)
Python 3.11.7 (main, Feb 22 2024, 17:25:45) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)] on linux
Houdini 20.5.278 hou module imported.
Type "help", "copyright", "credits" or "license" for more information.

Please let me know if this is all good - so I can put this terrifying issue to bed 😭

[sabjorn]

@aaronsmithtv doesn't seem to work on my side. sesictrl still segfaults. I can run:

sesictrl login --help

without it failing but running on it with or without the login credentials causes a segfault

[aaronsmithtv]

Are you sure that you're using the latest image (I updated it a few hours ago), as well as the exact hserver format I added above?

The segfault is actually not caused by the sesictrl login process itself, it is caused by the error message displayed when you've failed to log in. That's the reason that the updated flags work, because you're only getting the segfault when login fails.

I've tried it on a couple of different machines, and had others test it, so it definitely works from my perspective. Here's my full log on an attempt I made just now in a fresh environment:

docker run -it aaronsmithtv/hbuild:latest

root@9efb6c4fdb3d:/# hserver --clientid "..." --clientsecret "..." --host "https://www.sidefx.com/license/sesinetd"
sh: 1: systemctl: not found
root@9efb6c4fdb3d:/# sesictrl login
You are logged in with aaronsmithtv (account: 51443, company: aaronsmith.tv)
root@9efb6c4fdb3d:/# 

[sabjorn]

image id is:

aaronsmithtv/hbuild                20.5.278-base   ba337bf91887   4 hours ago      6.5GB

here is the output of the commands:

root@2047e841c04d:/#  hserver --clientid $SIDEFX_CLIENT --clientsecret $SIDEFX_SECRET --host "https://www.sidefx.com/license/sesinetd"
sh: 1: systemctl: not found
root@2047e841c04d:/# sesictrl login
You are not logged into the license server. [Error L01]
20: Fatal error: Segmentation fault (sent by pid -24)
-- TRACEBACK BEGIN --
Traceback from sesictrl 20.5.278 (Compiled on linux-x86_64-gcc9.3):
malloc_usable_size <sesictrl>
malloc_usable_size <sesictrl>
malloc_usable_size <sesictrl>
malloc_usable_size <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
mremap <sesictrl>
__libc_init_first <libc.so.6>
__libc_start_main <libc.so.6>
mremap <sesictrl>
-- TRACEBACK END --

[aaronsmithtv]

Hmm, I would say then that the likeliest problem is that $SIDEFX_CLIENT and $SIDEFX_SECRET are not correctly set.

Maybe just try filling the args directly with the credentials from the API manager itself: https://www.sidefx.com/oauth2/applications/

Make sure that the credentials are of the type authorization-code.

[sabjorn]

@aaronsmithtv ENV values were correct. BUT looks like the authorization-code is the issue. Previously this was set to client credentials type. Odd that Houdini has changed the compatibility of this...

[aaronsmithtv]

Aha ok great! I'm glad you managed to find a solution. It does appear that API login has changed quite substantially since the update - hopefully SideFX can make the process a little easier to understand :)

Closing this now, the new args should be reflected in the README updates of #12.

[sabjorn]

Ty!