LineageOS 21 and detailed instructions on adding a flavor

[erahhal]

I'm looking to use Robotnix to build LineageOS 21 for the Pixel 7 Pro. I read the wiki, and have been parsing through the code base, and still don't have a sense as to how deep the rabbit hole goes. I'm assuming that it's not trivial to add a new version of Lineage, otherwise it probably would have been done already - is that an accurate assessment? Here's what I've done so far:

• Added a flavors/lineageos/lineage-21.0 folder, copied from lineage-20.0
• ran nix develop in the flavors path
• ran ./update.sh lineage-21.0

it then fails with:

Traceback (most recent call last):
  File "/var/tmp/tmpi5n4xlmh/.repo/repo/main.py", line 56, in <module>
    from subcmds.version import Version
  File "/var/tmp/tmpi5n4xlmh/.repo/repo/subcmds/__init__.py", line 35, in <module>
    mod = __import__(__name__,
          ^^^^^^^^^^^^^^^^^^^^
  File "/var/tmp/tmpi5n4xlmh/.repo/repo/subcmds/help.py", line 20, in <module>
    from formatter import AbstractFormatter, DumbWriter
ModuleNotFoundError: No module named 'formatter'
Traceback (most recent call last):
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/../../scripts/mk_repo_file.py", line 294, in <module>
    main()
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/../../scripts/mk_repo_file.py", line 281, in main
    make_repo_file(args.url, args.ref, ref_type, prev_data,
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/../../scripts/mk_repo_file.py", line 106, in make_repo_file
    subprocess.check_call([
  File "/nix/store/yvhwsfbh4bc99vfvwpaa70m4yng4pvpz-python3-3.11.8/lib/python3.11/subprocess.py", line 413, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['repo', 'init', '--manifest-url=https://github.com/LineageOS/android', '--manifest-branch=refs/heads/lineage-21.0', '--quiet', '--repo-url=https://github.com/danielfullmer/tools_repo', '--repo-rev=9ecb9713ee5adba95120acbc0bfef1c77b02637f', '--no-repo-verify', '--depth=1']' returned non-zero exit status 1.

device-metdata.json does seem to be updated, but that's it. At the moment, I have no idea whether I'm 3% of the way to adding LineageOS 21.0, or 50%. I'm guess it's more like 3%.

Is there a detailed reference on the steps needed to do this? Is someone already working on this and I can either abort my efforts or help out?

[Atemu]

It's not trivial but also not unreasonably hard. It roughly goes like this:

1. Make robotnix aware of the existence of another version
2. Get new version's sources
3. Update patches
4. Make modules that broke compatible (usually also involves updating patches)

Take a look at https://github.com/nix-community/robotnix/pull/213/ and specifically https://github.com/nix-community/robotnix/pull/213/commits/11e02c0ce2452b3822a2f8695ac600b6e245ac9a.

Having it documented properly would be great though.

Don't copy over the directory for the sources though; you want to start fresh.

As for the error, it appears that the repo command failed for some reason. Run it manually and figure out why.

[Atemu]

Is someone already working on this and I can either abort my efforts or help out?

Missed this part: I'm not working on it and I am also not aware of anyone else who could be working on it. @eyJhb might be but I don't think they are.

[erahhal]

Thanks! I'll look at that commit today.

Regarding the error, this is what I see when I run the command directly:

repo init --manifest-url=https://github.com/LineageOS/android --manifest-branch=refs/heads/lineage-21.0/ --quiet --repo-url=https://github.com/danielfullmer/tools_repo --repo-rev=9ecb9713ee5adba95120acbc0bfef1c77b02637f --no-repo-verify --depth=1

Traceback (most recent call last):                                                                                                                                                                                                  
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/.repo/repo/main.py", line 56, in <module>                                                                                                                              
    from subcmds.version import Version                                                                                                                                                                                             
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/.repo/repo/subcmds/__init__.py", line 35, in <module>                                                                                                                  
    mod = __import__(__name__,                                                                                                                                                                                                      
          ^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                      
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/.repo/repo/subcmds/help.py", line 20, in <module>                                                                                                                      
    from formatter import AbstractFormatter, DumbWriter                                                                                                                                                                             
ModuleNotFoundError: No module named 'formatter' 

A quick search does reveal others with a similar problem. Looks like this module was removed with Python 3.10.

[Atemu]

That might be fallout from https://github.com/nix-community/robotnix/pull/231

You could revert that merge (or check out its first parent) enter the nix shell and then cancel the revert/go back to your branch for a quick workaround.

[erahhal]

Your workaround did the trick.

[erahhal]

I added changes similar to #213. Now after running update, It runs for a while fetching information from dozens of external repos, then it starts listing out code names for which a branch doesn't exist, then it attempts to authenticate with github, after which it fails. It seems that one of the repos has disappeared or gone private. Here's a truncated log:

.
.
.
Fetching information for https://android.googlesource.com/platform/tools/acloud a6e451eeb517c80702362f457476de5d74fef628
Fetching information for https://github.com/LineageOS/android_vendor_qcom_opensource_thermal-engine 0815f34094ebc952c8e1dd3619e384c7f9d1195b
Fetching information for https://github.com/LineageOS/android_vendor_qcom_opensource_usb 737d94e59c0cddb46369f17dd7bb83189facb913
Fetching information for https://github.com/LineageOS/android_vendor_qcom_opensource_vibrator a4af86a19783eb5e86c07a72ac0ece5b6a77fa4c
SKIP: lineage-21.0 branch does not exist for DRG
SKIP: lineage-21.0 branch does not exist for FP2
SKIP: lineage-21.0 branch does not exist for FP3
.
.
.
SKIP: lineage-21.0 branch does not exist for z3c
SKIP: lineage-21.0 branch does not exist for zippo
SKIP: lineage-21.0 branch does not exist for zl1
SKIP: lineage-21.0 branch does not exist for https://github.com/TheMuppets/proprietary_vendor_bq

**ATTEMPTS TO AUTHENTICATE HERE**

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/TheMuppets/proprietary_vendor_nothing/'
Traceback (most recent call last):
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/./update_device_dirs.py", line 255, in <module>
    main()
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/./update_device_dirs.py", line 249, in main
    fetch_vendor_dirs(metadata, "https://github.com/TheMuppets", args.branch, true_branch,
  File "/home/erahhal/Code-vendor/robotnix/flavors/lineageos/./update_device_dirs.py", line 199, in fetch_vendor_dirs
    refs = ls_remote(url)
  File "/nix/store/zdxrnv6a2l467pdpgyqarjbqqphn326w-scripts/robotnix_common.py", line 87, in ls_remote
    remote_info = subprocess.check_output(["git", "ls-remote", url]).decode()
  File "/nix/store/5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/nix/store/5bh6rpya1ar6l49vrhx1rg58dsa42906-python3-3.9.6/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['git', 'ls-remote', 'https://github.com/TheMuppets/proprietary_vendor_nothing']' returned non-zero exit status 128.

[erahhal]

I think it's not finding a lineage-21 branch in https://github.com/TheMuppets/proprietary_vendor_bq, and then constructing a URL for a git repo with an empty value nothing, https://github.com/TheMuppets/proprietary_vendor_nothing/, which github then tries to authenticate because it might be a private repo.

I'll see if it's possible to fork the first repo, add the branch/tag, and use it.

[erahhal]

Seems that https://github.com/TheMuppets/proprietary_vendor_bq reference is generated automatically and not manually specified.

[Atemu]

Github doesn't differentiate between private repo and 404 in order to not reveal the existence of private repos to the public. If you're being asked for creds for a repo under the LOS org, it's likely just a 404.

404s happen because the script is wrong and/or there is not strictly enforced convention it could follow. We construct repo paths based on assumptions that are true for most devices.

There's a bunch of workarounds for devices already, just add more to the pile...

I've recently started to simply ignore entire devices that do not follow the conventions of all the other devices.

[erahhal]

I'm not sure where to begin to address this. Can't find what is pulling in this https://github.com/TheMuppets/proprietary_vendor_bq repo. Searching github, it seems that it's only referenced in the robotnix vendor-dirs.json files. Were those edited manually or automatically generated? If the latter, what is being used to populate them?

[Atemu]

The vendor- aswell as device-dirs are generated by flavors/lineageos/update_device_dirs.py.

proprietary_vendor_<vendorname> is the legacy naming used by older versions of LOS. In LOS20, it's proprietary_vendor_<vendorname>_<devicename> now.

You'll therefore probably have to adjust this:

https://github.com/nix-community/robotnix/blob/178452589a25005f1d65b06aad8ab02b85a7f789/flavors/lineageos/update_device_dirs.py#L144

            if branch in [ 'lineage-20.0', 'lineage-21.0' ]:

Or some cleaner code that basically does lib.versionAtLeast.

Also please ensure you haven't copied any of the JSONs.

[erahhal]

I just created a lineage-21.0 directory, and didn't copy anything into it.

It indeed completed without fail this time! But still a long list of SKIP messages, e.g.

SKIP: mermaid is not available for lineage-21.0

And in the end only the repo.json file is in the lineage-21.0 folder, no device-dirs.json or vendor-dirs.json.

I'll try to piece through the code to see how it generates those files.

[erahhal]

I also made a change to how it calculates true_branch at line 231. Seems to be running longer now, perhaps it will work this time...

[erahhal]

Ok, device-dirs.json now exists, woohoo!

No vendor-dirs.json tho, debugging that now.

[erahhal]

Looks like they migrated from master to main at https://github.com/LineageOS/hudson.

Running again with the update.

BTW, is there a chat room, e.g. matrix or IRC or whatever where people keep in touch re: this project? Or is the volume low enough to keep it on Github?

[ajs124]

@erahhal there's https://matrix.to/#/#robotnix:nixos.org

[erahhal]

Thank you, I'll join that room. I'm in the process of updating the patches now.

[jaen]

I've also came across the ModuleNotFoundError: No module named 'formatter' error, rebasing danielfullmer/tools_repo on current upstream version did the trick — here's the diff if you're interested: https://github.com/jaen/tools_repo/pull/1.

No idea if it works properly after the rebase yet, though.