Setting environment variables on android devices

[jamescrosswell]

I'm trying to do something like this:

public class MyTests{
    [SkippableFact]
    public async Some_Flaky_Test()
    {
        Skip.If(TestEnvironment.IsGitHubActions, "This test is flaky on GitHub Actions");
        // ...
    }
}

public static class TestEnvironment
{
    public static bool IsGitHubActions
    {
        get
        {
            var isGitHubActions = Environment.GetEnvironmentVariable("GITHUB_ACTIONS");
            return isGitHubActions?.Equals("true", StringComparison.OrdinalIgnoreCase) == true;
        }
    }
}

However the test is still being run in CI (using xharness android test). I suspect although the CI runner (macos) has the GITHUB_ACTIONS environment variable set, this doesn't automatically get set on the android device where the tests are being run.

I see xharness apple test accepts a --set-env argument, but I can't find any equivalent for xharness android test.

Is there a way to propagate/set environment variables on android devices before unit tests are run?

[ivanpovazan]

Hello @jamescrosswell,

In essence you need to use instrumentation to achieve this, which can be done through the following:

1. In your test app you need to implement the instrumentation class, e.g.: https://github.com/mattleibow/DeviceRunners/blob/cba7644e07b305ba64dc930b01c3eee55ef2b93d/src/DeviceRunners.XHarness.Maui/Platforms/Android/XHarnessInstrumentation.cs#L8

which accepts arguments and stores them OnCreate (you can override e.g., OnStart and set whatever is stored in Arguments to the environment)

2. When calling xharness instruct it to use your instrumentation class by including the following in your command line: --instrumentation devicerunners.xharness.maui.XHarnessInstrumentation which can accept arguments by including --arg=key=value and pass them to the instrumentation (more info at: https://github.com/dotnet/xharness/blob/2b6293f14cf55326dde74b1364d12c579a8bd569/src/Microsoft.DotNet.XHarness.CLI/CommandArguments/Android/Arguments/InstrumentationArguments.cs#L13)

   All together it would look something like:

   xharness android test --app your.apk --package-name com.companyname.yourapp --instrumentation devicerunners.xharness.maui.XHarnessInstrumentation --output-directory ./tmp --verbosity=Debug --arg=key1=value1 --arg=key2=value2

   Hope this helps!

   PS @mattleibow can add more context if needed as he is the maintainer of https://github.com/mattleibow/DeviceRunners

[mattleibow]

OK, maybe I can add some automatic envvar setting if you were to do --arg=ENV_envvarname=envvarral just to make things easier to set and less code to write.

[jamescrosswell]

OK, maybe I can add some automatic envvar setting if you were to do --arg=ENV_envvarname=envvarral just to make things easier to set and less code to write.

@mattleibow

Ideally it could be done the same way it's done for the apple tests: xharness android test --set-env key=value

... but I guess that requires a change to xharness (not just the DeviceRunners package).

Failing that, the above would be fantastic!

[jamescrosswell]

In essence you need to use instrumentation to achieve this

@ivanpovazan thanks - that gives me a possible work around.

TLDR; I also had to add <AndroidEnableMarshalMethods>false</AndroidEnableMarshalMethods> to the csproj file for my test app.

Detailed explanation

I tried with this class:

using Android.App;
using Android.Runtime;
using DeviceRunners.XHarness.Maui;
using Environment = System.Environment;

namespace Sentry.Maui.Device.TestApp;

[Instrumentation(Name = "Sentry.Maui.Device.TestApp.SentryInstrumentation")]
public class SentryInstrumentation : XHarnessInstrumentation
{
    protected SentryInstrumentation(IntPtr handle, JniHandleOwnership ownership)
        : base(handle, ownership)
    {
    }

    public override void OnStart()
    {
        try
        {
            Console.WriteLine("Parsing instrumentation arguments");
            if (IsGitHubActions)
            {
                Console.WriteLine("CI build detected - setting environment variables for CI on the device");
                Environment.SetEnvironmentVariable("GITHUB_ACTIONS", "true");
            }
            else
            {
                Console.WriteLine("CI build not detected - no environment variables set");
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e);
        }

        base.OnStart();
    }

    private bool IsGitHubActions
    {
        get
        {
            if (Arguments is { } bundle && bundle.KeySet() is { } keySet && keySet.Contains("IsGitHubActions"))
            {
                return bundle.GetString("IsGitHubActions") == "true";
            }

            return false;
        }
    }
}

The logs indicated that the instrumentation class was used but the test app crashes when run in xharness 😢

info: Running instrumentation class Sentry.Maui.Device.TestApp.SentryInstrumentation took 0.4560827 seconds
info: Short message:
      Process crashed.
fail: No value for 'return-code' provided in instrumentation result. This may indicate a crashed test (see log)

I played around a bit more and realised that this would not crash:

            if (Arguments is { } bundle && bundle.KeySet() is { } keySet && keySet.Contains("IsGitHubActions"))
            {
                return true;
            }

But this would:

            if (Arguments is { } bundle && bundle.KeySet() is { } keySet && keySet.Contains("IsGitHubActions"))
            {
                return bundle.GetString("IsGitHubActions") == "true";
            }

So any attempt to read the actual value of an argument from the bundle causes a crash.

In the adb logs I found this, just before the crash:

/__w/1/s/src/mono/mono/metadata/icall.c:6092, condition `!only_unmanaged_callers_only' not met

Googling that led me to this comment... and indeed adding <AndroidEnableMarshalMethods>false</AndroidEnableMarshalMethods> to the csproj file for the test app works around the problem.

@mattleibow in summary, if you could make this less painful for people, I think that'd be worth it. I think I only got the above working through sheer force of will and a bit of luck!

[ivanpovazan]

Ideally it could be done the same way it's done for the apple tests: xharness android test --set-env key=value

There is a difference there, as --arg can be used, and is used, for different purposes, not just setting environment variables. The extension point is the Instrumentation class which gives you freedom to handle the passed arguments as desired.

At the moment, we don't plan to change this behavior on the xharness side. We might consider it in the future to improve user/developer experience, but currently it is out of scope.

If I understood correctly there is nothing blocking you from xharness side and with that, I will close this issue as completed. Please feel free to reopen it, or open a new one if you experience any other issues with our tooling. Thanks for understanding.