global remote execution mode is broken for bootstrap stage

[Eric-Arellano]

https://github.com/pantsbuild/pants/pull/17135 needs to run in local execution mode when the bootstrap stage is happening. Otherwise we get:

Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/ericarellano/code/pants/src/python/pants/bin/pants_loader.py", line 127, in <module>
    main()
  File "/Users/ericarellano/code/pants/src/python/pants/bin/pants_loader.py", line 123, in main
    PantsLoader.main()
  File "/Users/ericarellano/code/pants/src/python/pants/bin/pants_loader.py", line 110, in main
    cls.run_default_entrypoint()
  File "/Users/ericarellano/code/pants/src/python/pants/bin/pants_loader.py", line 92, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/Users/ericarellano/code/pants/src/python/pants/bin/pants_runner.py", line 99, in run
    runner = LocalPantsRunner.create(
  File "/Users/ericarellano/code/pants/src/python/pants/bin/local_pants_runner.py", line 127, in create
    build_config = options_initializer.build_config(options_bootstrapper, env)
  File "/Users/ericarellano/code/pants/src/python/pants/init/options_initializer.py", line 111, in build_config
    return _initialize_build_configuration(self._plugin_resolver, options_bootstrapper, env)
  File "/Users/ericarellano/code/pants/src/python/pants/init/options_initializer.py", line 48, in _initialize_build_configuration
    working_set = plugin_resolver.resolve(options_bootstrapper, env)
  File "/Users/ericarellano/code/pants/src/python/pants/init/plugin_resolver.py", line 137, in resolve
    for resolved_plugin_location in self._resolve_plugins(
  File "/Users/ericarellano/code/pants/src/python/pants/init/plugin_resolver.py", line 164, in _resolve_plugins
    session.product_request(ResolvedPluginDistributions, [params])[0],
  File "/Users/ericarellano/code/pants/src/python/pants/engine/internals/scheduler.py", line 561, in product_request
    return self.execute(request)
  File "/Users/ericarellano/code/pants/src/python/pants/engine/internals/scheduler.py", line 505, in execute
    self._raise_on_error([t for _, t in throws])
  File "/Users/ericarellano/code/pants/src/python/pants/engine/internals/scheduler.py", line 489, in _raise_on_error
    raise ExecutionError(

Exception message: 1 Exception encountered:

Engine traceback:
  in Prepare environment for running PEXes
  in Scheduling: Extract environment variables from the remote execution environment

Exception: Failed to execute: Process {
    argv: [
        "env",
        "-0",
    ],
    env: {},
    working_directory: None,
    input_digests: InputDigests {
        complete: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>,
                size_bytes: 0,
            },
            tree: "Some(..)",
        },
        nailgun: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>,
                size_bytes: 0,
            },
            tree: "Some(..)",
        },
        input_files: DirectoryDigest {
            digest: Digest {
                hash: Fingerprint<e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855>,
                size_bytes: 0,
            },
            tree: "Some(..)",
        },
        immutable_inputs: {},
        use_nailgun: {},
    },
    output_files: {},
    output_directories: {},
    timeout: None,
    execution_slot_variable: None,
    concurrency_available: 0,
    description: "Extract environment variables from the remote execution environment",
    level: Debug,
    append_only_caches: {},
    jdk_home: None,
    platform: Linux_x86_64,
    cache_scope: Successful,
    execution_strategy: RemoteExecution(
        [],
    ),
    remote_cache_speculation_delay: 0ns,
}

Error launching process: Os { code: 2, kind: NotFound, message: "No such file or directory" }

I'm not sure how to know it's the bootstrap stage.

[stuhood]

"Bootstrap" uses a BootstrapScheduler, which is configured a particular way: https://github.com/pantsbuild/pants/blob/f16e9d0c57295ec48660a2350e461b891422821e/src/python/pants/init/options_initializer.py#L64-L82 ... it should always fully disable remote execution due to the DynamicRemoteOptions.disabled() line there.

[Eric-Arellano]

@stuhood it looks like that only impacts our Rust values, but it does not overwrite the values of OptionsBootstrapper used here:

https://github.com/pantsbuild/pants/blob/73f97f75a31c922d4380e7f4186cd05b77f09b0f/src/python/pants/init/plugin_resolver.py#L146-L166

options_bootstrapper.get_bootstrap_options().for_global_scope().remote_execution in this method returns True during the plugin resolution stage.

This is an issue because we now determine whether to use RE from PyProcessConfigFromEnvironment, whereas before it was a global toggle controlled by Rust.

I think we have two fixes:

1. Trick OptionsBootstrapper -> GlobalOptions into disabling all the dynamic remote options, e.g. add OptionsBootstrapper.remote_options_disabled()
2. Or, have a way to know that the session is the plugin resolver, and then update our environment-specific rules to handle it.

Suggestions?

[stuhood]

There is a pattern of adding static resolve methods to GlobalOptions which take the inputs needed to compute a value, rather than accessing it directly: https://github.com/pantsbuild/pants/blob/be00ed54ca4d68539c5d840bf2381b33603d747a/src/python/pants/option/global_options.py#L1838-L1919 ... but they're used purely by convention. In this case that might look like calling: GlobalOptions.resolve_remote_execution(global_options, dynamic_remote_options) in a @rule.

There isn't a way to "overlay" the DynamicRemoteOptions onto the options bootstrapper... but you could think of them as another options "Rank"... "DYNAMIC_OPTIONS" or "PLUGIN_OPTIONS" or something: https://github.com/pantsbuild/pants/blob/be00ed54ca4d68539c5d840bf2381b33603d747a/src/python/pants/option/ranked_value.py#L12-L20 ... and then have a facility for plugins to add options? ...pretty advanced.

A hacky way to accomplish something similar would be to essentially convert the DynamicRemoteOptions back into CLI args, and append those during bootstrap. Oy.

[Eric-Arellano]

Thanks for the feedback!

In this case that might look like calling: GlobalOptions.resolve_remote_execution(global_options, dynamic_remote_options) in a @rule.

The issue is we would need in the rule to still know whether we're in the plugin resolver or not. Generally, we'd want to use DynamicRemoteOptions as is. But in same cases, we want to use its .disabled() variant. Now, that could be injected into the graph!

But at that point, maybe it's easier to have some other sentinel mechanism so that these 3 relevant rules can check if they're in the plugin resolver and change their logic accordingly?

[Eric-Arellano]

But at that point, maybe it's easier to have some other sentinel mechanism so that these 3 relevant rules can check if they're in the plugin resolver and change their logic accordingly?

@stuhood thoughts on how we could know inside platform_rules.py that it's the plugin resolver session?

[stuhood]

Now, that could be injected into the graph!

Anything can be injected as a singleton without a lot of fanfare in EngineInitializer.setup_graph_extended... you could pass a wrapped boolean, pass the entire DynamicRemoteOptions in and then use it to call static methods, etc.

[Eric-Arellano]

@stuhood it looks like this is still an issue, right? I thought I had seen you fix something related, but can't find it.