[Build] mach boostrap fails with: Exception: python/mozboot/**/*android* did not match anything

yuripourre commented 4 months ago

Configuration

Wolvic version: master Wolvic build ID: -

Hardware: Dell Inspiron 15 7000 / Fedora 38

Steps to Reproduce

Follow the Developer workflow
Download the firefox-$VERSION.source.tar.xz
Apply the patches
Try to run ./mach bootstrap

Current Behavior

Build fails with the following error:

Exception: python/mozboot/**/*android* did not match anything

Expected Behavior

Command would succeed

Possible Solution

Context

Error Logs and Stack Traces

``` Note on Artifact Mode: Artifact builds download prebuilt C++ components rather than building them locally. Artifact builds are faster! Artifact builds are recommended for people working on Firefox or Firefox for Android frontends, or the GeckoView Java API. They are unsuitable for those working on C++ code. For more information see: https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html. Please choose the version of Firefox you want to build (see note above): 1. Firefox for Desktop Artifact Mode [default] 2. Firefox for Desktop 3. GeckoView/Firefox for Android Artifact Mode 4. GeckoView/Firefox for Android 5. SpiderMonkey JavaScript engine Your choice: 4 Requirement already satisfied: glean-sdk==55.0.0 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (55.0.0) Requirement already satisfied: semver>=2.13.0 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean-sdk==55.0.0) (3.0.2) Requirement already satisfied: glean_parser~=10.0 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean-sdk==55.0.0) (10.0.3) Requirement already satisfied: appdirs>=1.4 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean_parser~=10.0->glean-sdk==55.0.0) (1.4.4) Requirement already satisfied: Click>=7 in ./third_party/python/click (from glean_parser~=10.0->glean-sdk==55.0.0) (8.1.6) Requirement already satisfied: diskcache>=4 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean_parser~=10.0->glean-sdk==55.0.0) (5.6.3) Requirement already satisfied: Jinja2>=2.10.1 in ./third_party/python/Jinja2 (from glean_parser~=10.0->glean-sdk==55.0.0) (3.1.2) Requirement already satisfied: jsonschema>=3.0.2 in ./third_party/python/jsonschema (from glean_parser~=10.0->glean-sdk==55.0.0) (4.17.3) Requirement already satisfied: PyYAML>=5.3.1 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (from glean_parser~=10.0->glean-sdk==55.0.0) (6.0.1) Requirement already satisfied: MarkupSafe>=2.0 in ./third_party/python/MarkupSafe/src (from Jinja2>=2.10.1->glean_parser~=10.0->glean-sdk==55.0.0) (2.0.1) Requirement already satisfied: attrs>=17.4.0 in ./third_party/python/attrs (from jsonschema>=3.0.2->glean_parser~=10.0->glean-sdk==55.0.0) (23.1.0) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in ./third_party/python/pyrsistent (from jsonschema>=3.0.2->glean_parser~=10.0->glean-sdk==55.0.0) (0.16.0) Requirement already satisfied: six in ./third_party/python/six (from pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0->jsonschema>=3.0.2->glean_parser~=10.0->glean-sdk==55.0.0) (1.16.0) Requirement already satisfied: psutil<=5.9.4,>=5.4.2 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (5.9.4) Requirement already satisfied: zstandard<=0.22.0,>=0.11.1 in /home/user/.mozbuild/srcdirs/firefox-121.0.1-d5946a825706/_virtualenvs/mach/lib/python3.11/site-packages (0.22.0) Looks like you have the correct version of the Android NDK installed at: /home/user/.mozbuild/android-ndk-r23c Looks like you have the Android SDK installed at: /home/user/.mozbuild/android-sdk-linux We will install all required Android packages. We are now installing the following Android packages: emulator platform-tools build-tools;33.0.1 platforms;android-33 You may be prompted to agree to the Android license. You may see some of output as packages are downloaded and installed. Warning: Errors during XML parse: Warning: Additionally, the fallback loader failed to parse the XML. [=======================================] 100% Computing updates... Looks like you have the correct version of the Android NDK installed at: /home/user/.mozbuild/android-ndk-r23c Looks like you have the Android SDK installed at: /home/user/.mozbuild/android-sdk-linux We will install all required Android packages. We are now installing the following Android packages: emulator system-images;android-24;default;x86_64 You may be prompted to agree to the Android license. You may see some of output as packages are downloaded and installed. Warning: Errors during XML parse: Warning: Additionally, the fallback loader failed to parse the XML. [=======================================] 100% Computing updates... Looks like you have the correct version of the Android NDK installed at: /home/user/.mozbuild/android-ndk-r23c Looks like you have the Android SDK installed at: /home/user/.mozbuild/android-sdk-linux We will install all required Android packages. We are now installing the following Android packages: emulator system-images;android-24;default;armeabi-v7a You may be prompted to agree to the Android license. You may see some of output as packages are downloaded and installed. Warning: Errors during XML parse: Warning: Additionally, the fallback loader failed to parse the XML. [=======================================] 100% Computing updates... Your version of Rust (1.78.0) is new enough. Rust supports aarch64-linux-android, i686-linux-android, thumbv7neon-linux-androideabi, x86_64-linux-android, x86_64-unknown-linux-gnu targets. Error running mach: mach --log-no-times artifact toolchain --from-build toolchain-linux64-clang The error occurred in code that was called by the mach command. This is either a bug in the called code itself or in the way that mach is calling it. You can invoke ``./mach busted`` to check if this issue is already on file. If it isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is misbehaving, you can also inspect the dependencies of bug 1543241. If filing a bug, please include the full output of mach, including this error message. The details of the failure are as follows: Exception: python/mozboot/**/*android* did not match anything File "/home/user/git/firefox-121.0.1/python/mozbuild/mozbuild/artifact_commands.py", line 393, in artifact_toolchain tasks = toolchain_task_definitions() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/python/mozbuild/mozbuild/toolchains.py", line 20, in toolchain_task_definitions toolchains = load_tasks_for_kind(params, "toolchain", root_dir=root_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 449, in load_tasks_for_kind for task in tgg.full_task_set ^^^^^^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 160, in full_task_set return self._run_until("full_task_set") ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 425, in _run_until k, v = next(self._run) ^^^^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 311, in _run new_tasks = kind.load_tasks( ^^^^^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in load_tasks tasks = [ ^ File "/home/user/git/firefox-121.0.1/third_party/python/taskcluster_taskgraph/taskgraph/generator.py", line 76, in tasks = [ ^ File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 2219, in check_run_task_caches for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 2129, in check_task_dependencies for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 2115, in check_task_identifiers for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 2096, in chain_of_trust for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1899, in build_task for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1872, in set_task_and_artifact_expiry for job in jobs: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1845, in try_task_config_routes for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1832, in try_task_config_chemspill_prio yield from tasks File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1812, in try_task_config_env yield from tasks File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1779, in add_index_routes for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1574, in validate for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1552, in task_name_from_label for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1542, in setup_raptor for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1486, in set_defaults for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/task.py", line 1462, in set_implementation for task in tasks: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/cached_tasks.py", line 65, in cache_task for task in order_tasks(config, tasks): File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/cached_tasks.py", line 20, in order_tasks pending = deque(tasks) ^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 282, in use_fetches jobs = list(jobs) ^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 247, in use_system_python for job in jobs: File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 228, in make_task_description configure_taskdesc_for_run( File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/__init__.py", line 507, in configure_taskdesc_for_run func(config, job, taskdesc) File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/toolchain.py", line 247, in docker_worker_toolchain common_toolchain(config, job, taskdesc, is_docker=True) File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/toolchain.py", line 170, in common_toolchain digest_data = get_digest_data(config, run, taskdesc) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/transforms/job/toolchain.py", line 82, in get_digest_data data = [hash_paths(GECKO, files)] ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/git/firefox-121.0.1/taskcluster/gecko_taskgraph/util/hash.py", line 57, in hash_paths raise Exception("%s did not match anything" % pattern) ```

svillar commented 4 months ago

Could you please share the mozbuild file you're using?

svillar commented 4 months ago

Could you please share the mozbuild file you're using?

Also did mach bootstrap went well with no issues?

yuripourre commented 4 months ago

Could you please share the mozbuild file you're using?

Thanks for the reply! Sure, I can share but I haven't touched it. It would be the same as the one in the gecko-dev repo.

Also did mach bootstrap went well with no issues?

No, actually the error happens exactly when I run ./mach bootstrap.

svillar commented 4 months ago

Could you please share the mozbuild file you're using?

Thanks for the reply! Sure, I can share but I haven't touched it. It would be the same as the one in the gecko-dev repo.

Also did mach bootstrap went well with no issues?

No, actually the error happens exactly when I run ./mach bootstrap.

Right, again the problem of bootstrap ever changing the dependencies it downloads depending on the date you execute it :(. I am not sure there is an easy fix for that. I don't want to change my dev environment now that we're in the middle of a release but I can try to reproduce the issue after that

yuripourre commented 4 months ago

@svillar Thank you, I appreciate your efforts. I will try some ideas this weekend and I will update here if I have some progress.

weller0 commented 4 months ago

The following methods can be used to build Download gecko124.0.1 and then build the environment in gecko124.0.1

./mach bootstrap

Switch to gecko121.0.1 again to build

./mach build

yuripourre commented 4 months ago

Thank you @weller0! That worked! I could proceed after using your suggestion.

I had more errors, fixed some and currently struggling with libclang_rt.builtins-aarch64-android.a not found. I am trying to compile the compiler-rt but having issues in Fedora, apparently some internal packages cannot be found (Fedora problems, no worries).

I think we consider it closed now. Thank you very much!

cyberfoxmeow commented 2 months ago

can refer to : https://bugzilla.mozilla.org/show_bug.cgi?id=1906930

Igalia / wolvic