search

redhat-developer / dotnet-regular-tests

.NET Core tests for .NET Bunny (RHEL & Fedora RPM tests)
MIT License
8 stars 13 forks source link

`lttng` test fails on Fedora #202

Open omajid opened 2 years ago

omajid commented 2 years ago

See https://github.com/redhat-developer/dotnet-regular-tests/runs/8078697017?check_suite_focus=true:

# Standard Output:
Executing /__w/dotnet-regular-tests/dotnet-regular-tests/lttng/test.sh with arguments 6.0.8 in working directory /__w/dotnet-regular-tests/dotnet-regular-tests/lttng
== Starting lttng session
Session my-session created.
Traces will be output to /tmp/my-session/trace
ust context vpid added to all channels
ust event DotNETRuntime:RuntimeInformationStart created in channel channel0
Tracing started for session my-session
== Creating new console application

Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.108

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
The template "Console App" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /tmp/my-session/console/console.csproj...
  Determining projects to restore...
  Restored /tmp/my-session/console/console.csproj (in 186 ms).
Restore succeeded.

== Ending lttng session
Waiting for data availability
Tracing stopped for session my-session
Destroying session my-session..
Session my-session destroyed
Exit Code: 1
# Standard Error:
Error:  Cannot open any trace for reading.

Error:  opening trace "�$�5" for reading.

Error:  none of the specified trace paths could be opened.
omajid commented 2 years ago

@tmds do you have some cycles to dig into this?

tmds commented 2 years ago

Does this fail anywhere other than Fedora Rawhide?

omajid commented 2 years ago

37 too, I think. Doesn't fail on 35 or 36. I haven't tested on any other distribution family.

tmds commented 2 years ago

Unless this is urgent, I'll take a look when I upgrade my laptop to Fedora 37, which will happen in the next one or two weeks.

tmds commented 2 years ago

I ran this test on Fedora 37. There is nothing wrong with the .NET side of things: lttng-sessiond receives the a DotNETRuntime:RuntimeInformationStart event.

lttng-sessiond also creates the trace folder at the expected location of /tmp/my-session/trace/ust/uid/1000/64-bit.

babeltrace is not happy with what's in that folder, and spits out the error messages we see in the log:

Error:  Cannot open any trace for reading.

Error:  opening trace "�$�5" for reading.

Error:  none of the specified trace paths could be opened.

The version numbers on the packages seem to indicate not much changed between f36 and f37:

lttng-tools                     2.13.4-1.fc36
lttng-ust                       2.13.1-2.fc36
babeltrace                      1.5.8-9.fc36

lttng-tools.x86_64              2.13.7-3.fc37                
lttng-ust.x86_64                2.13.3-3.fc37  
babeltrace.x86_64               1.5.8-12.fc37
tmds commented 1 year ago

Current state: test passes on Fedora 38, and fails with Fedora Rawhide (39).

tmds commented 1 year ago

The test is now crashing on Fedora 38:

: Executing /tmp/tmp.Dkr0O4tRps/dotnet-regular-tests/lttng/test.sh with arguments 8.0.0-rc.2.23418.14 in working directory /tmp/tmp.Dkr0O4tRps/dotnet-regular-tests/lttng
== Starting lttng session
Session my-session created.
Traces will be output to /tmp/my-session/trace
ust context vpid added to all channels
ust event DotNETRuntime:RuntimeInformationStart created in channel channel0
Tracing started for session my-session
== Creating new console application
/tmp/tmp.Dkr0O4tRps/dotnet-regular-tests/lttng/test.sh: line 46: 134211 Segmentation fault      (core dumped) dotnet new console -o $TEST_FOLDER/console
Process Exit Code: 139
ayakael commented 10 months ago

What is best practice for mitigating lttng-ust 2.13 and up issues? On Alpine we've been removing and touching libcoreclrtraceptprovider.so but I'm not sure that's the best approach anymore.

tmds commented 10 months ago

What is best practice for mitigating lttng-ust 2.13 and up issues?

I have not investigated this further.

I have not used LTTNG with .NET in a very long time, and no one else seems to have noticed except our CI...

Do you know what potentially is the root cause?

kiran-1992n commented 3 months ago

See https://github.com/redhat-developer/dotnet-regular-tests/runs/8078697017?check_suite_focus=true:

# Standard Output:
Executing /__w/dotnet-regular-tests/dotnet-regular-tests/lttng/test.sh with arguments 6.0.8 in working directory /__w/dotnet-regular-tests/dotnet-regular-tests/lttng
== Starting lttng session
Session my-session created.
Traces will be output to /tmp/my-session/trace
ust context vpid added to all channels
ust event DotNETRuntime:RuntimeInformationStart created in channel channel0
Tracing started for session my-session
== Creating new console application

Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.108

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
The template "Console App" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /tmp/my-session/console/console.csproj...
  Determining projects to restore...
  Restored /tmp/my-session/console/console.csproj (in 186 ms).
Restore succeeded.

== Ending lttng session
Waiting for data availability
Tracing stopped for session my-session
Destroying session my-session..
Session my-session destroyed
Exit Code: 1
# Standard Error:
Error:  Cannot open any trace for reading.

Error:  opening trace "�$�5" for reading.

Error:  none of the specified trace paths could be opened.

LTTNG is not generating trace files on RHEL 8.8. Getting same trace . babeltrace is not happy with what's in that folder - There are no trace files generated in the folder. Any Suggestions Appreciated.