Qonfused
commented
1 year ago Discord link for the original conversation: /r/Hackintosh Paradise - #coding
Open Qonfused opened 1 year ago
Qonfused
commented
1 year ago Discord link for the original conversation: /r/Hackintosh Paradise - #coding
vit9696
commented
1 year ago Hi @Qonfused! This is something I have always wanted to appear, and it is really great to see it emerge on your side. I am not great in YAML, but we used this format here and there, and it feels to me a good fit.
I think my personal suggestion would be to minimise the amount of dependencies and perhaps eventually upstream this to pip. This way one should more or less easily set it up and configure.
If you need me to answer OC-related questions or discuss something, I will do my best to provide the information. Also, it may be great to eventually have this package in github.com/acidanthera if you do not mind ^_^
Qonfused
commented
1 year ago Definitely some good suggestions there.
The YAML implementation in this library adds a superset of features to the existing YAML 1.2 spec, though any format supported by Python is easily amenable to support the same features. JSON for example works natively with Python and would also be supported. There are also shared parsing utilities that can make way for custom binary formats or schemas.
I also completely agree with your remark about dependency size. I've kept the dependency list for packaging and library consumption as slim as possible (pyproject.toml#dependencies); for the most part this falls under a subset of PIP's own vendored dependencies. The primary motivation is to aid in security audits for CI and publishing but also to minimize unexpected behavior when using tools like Cython or Pyinstaller.
On another note with PIP, the versioning system and methods used in this project extend Python's packaging library to additionally be compliant with the wider Semver standard (used by package managers like NPM). There are some tighter constraints that the PIP project has to cater to specifically for the Python ecosystem (i.e. PEP 440 and PEP 425). Though plenty of native abstractions and extensions of stdlib modules in this library can make their way into PIP or a future version of Python if there is interest.
For now, I'll work on merging the current PR and try to code-golf an MVP implementation before grappling with larger infrastructure needs. I also don't at all mind moving this project to the acidanthera org.
Qonfused
commented
1 year ago It's worth noting that the CLI is distributed as a package script runnable through PIP when the ocebuild library is installed. It would function the same as it would in development when normally ran through poetry run ocebuild; instead, you can just run ocebuild to invoke the CLI:
$ ocebuild --version
ocebuild-cli 0.0.0-devThe entrypoint for the library is also under the same name:
>>> from ocebuild import __version__
... print(__version__)
'0.0.0-dev'Though what I primarily had in mind for distribution was to release builds (using PyInstaller and/or Cython) with different arch/platform targets (i.e. 32/64 bit, arm/x86, windows/linux/macOS). Depending on how agnostic the platform-targeted builds are, it may also suffice to just build a single binary per platform instead. Very much open to ideas there.
Qonfused
commented
1 year ago Will want to solicit more feedback from users to make sure the project is well calibrated.
yordis
commented
1 year ago Sharing the learning from upgrading to the latest OpenCore (thank you so much @Qonfused),
Ideally, OpenCore suggests either copy add an entry to the config.plist with the version of OpenCore installed. Because I didn't track the version, knowing what version I was upgrading took a lot of work. I ended up redoing the entire thing from scratch.
Comparing Sample.plist and config.plist was practically impossible. Still trying to figure out the ideal implementation here, but it was a huge pain point.
Be able to have after installs scripts (such as checking for the video driver, or follow up potential tuning or ....)
I ended up adding multiple submodules and a bunch of make commands:
More and less what I ended up with thus far in terms of directory structure:
Ideally, the configuration for the drivers and whatnot could be simplified by bringing some details from the docs into a CLI, if possible.
And lastly, but to mention, give some love to the existing documentation.
Qonfused
commented
1 year ago Still cranking out a CI pipeline for builds, but for now I've uploaded some binaries in a GitHub pre-release for testing: https://github.com/Qonfused/OCE-Build/releases/tag/0.0.0-dev
Note that these binaries are versioned as 0.1.0-dev; that's to be expected.
Qonfused
commented
1 year ago Beyond the project's examples/ configs, I've also added an updated config in my own UX481FL repo for testing: https://github.com/Qonfused/ASUS-ZenBook-Duo-14-UX481-Hackintosh/tree/ocebuild-rewrite
This issue has been requested to solicit feedback on the OCE-Build project's direction and to discuss the project's roadmap. The proposal outlined in this issue is not final and is subject to change should this project mature or make its way into an upstream Acidanthera project. Please note that the current state of the project is very Alpha and is not yet intended for production use.
This project is not intended to replace existing configurations or tools used with OpenCore, but instead to supplement it with a more user-friendly format that is easier to version and manage with existing tools. Additionally, this project is intended to be used to support new or existing tools and workflows, where user feedback and feature requests will have the most impact in supporting.
Motivation
The OpenCore ecosystem has grown significantly over the past few years, and with it, the complexity of the configuration format and the number of tools that support it. This has led to a number of issues and user-desired features that are difficult for OpenCore maintainers to solve without introducing additional infrastructure or tooling.
This project aims to address these issues by introducing a supplemental configuration format and build system that is designed to be easily versioned and merged with existing tools like git. This project also introduces a build configuration format (expressing the structure of the EFI folder) that is designed to be easily written and maintained by the user. This is accompanied with a versioning system with support for incremental builds and versioning constraints that can be used to build new or existing projects or to patch individual components. These components serve to extend the capabilities of OpenCore and it's supporting ecosystem, as well as providing a foundation for new tools to be built on top of.
Additionally, versioning dependencies with a build configuration also allows for tracking down bugs easier by making the transition between release and debug versions (or prior versions) of dependencies turn-key. This is an essential aid to debugging and testing plugins (e.g. Kexts), as it allows for a user to easily switch between different versions of dependencies without having to manually source and download them. This also serves as a debugging tool to inform users and plugin maintainers of the exact versions of dependencies in a user's configuration.
This is especially useful for projects like OCLP that are more complex and could leverage this information to reduce the complexity of investigation and triage of new bugs or regressions. Other projects in the ecosystem can use this project's library to extend quality-assuring features that are agnostic to the configuration format without the burden of maintaining schemas or heuristics themselves.
To support user configuration, OpenCore uses a configuration format that is written in a binary format (plist) that is difficult to version and merge with conventional versioning tools. This is due to the fact that the configuration format is tightly coupled with the configuration schema, which is versioned separately from the user's configuration format. This makes it difficult to track changes between versions of the configuration format, as well as making it difficult to merge changes between different versions of the user's made changes. This format is however essential to the ecosystem it supports (macOS) and is the only format that is supported by the OpenCore bootloader. This makes it difficult to introduce an alternative configuration format without breaking compatibility with the bootloader, as well as making it difficult to introduce new features by existing tools. However, the binary format of the configuration format is essential to the bootloader and ecosystem, and must instead be converted to and from a declarative format that is much more easily versioned and incorporable with existing tools. To do this, a number of problems with the existing binary format must be addressed to resolve a user-maintainable configuration format without the need for replacing the existing configuration format used by OpenCore. ### i. Schema conflicts and configuration coupling A good way to contextualize schema and configuration coupling is with the config.plist file, as the [Sample.plist](https://github.com/acidanthera/OpenCorePkg/blob/master/Docs/Sample.plist) defaults and schema are written with its user-written configuration without distinction. Because of this, it's very difficult for a user to scan these configuration files and determine what properties have been changed from defaults, or which keys are missing from newer configuration schemas. This additionally makes undoing changes between different versions more difficult as changes are not forwards compatible, and backward compatibility is coupled with newer configuration schemas. I've written out a fairly detailed breakdown of methods and rationale for handling resolving schema conflicts [here](https://github.com/Qonfused/OCE-Build/issues/1) using existing tools via git. With git, however, changes are globbed on a line-by-line basis, which may implicitly depend on a globbed older plist schema even if only a single property was actually changed by the user. Because of this, changes are still coupled with the configuration schema in diffs (and still presents the same coupling issue with defaults). This use-case also falls outside of the scope of existing tools in the OpenCore ecosystem like ocvalidate (see https://github.com/acidanthera/bugtracker/issues/2236) and [OCConfigCompare](https://github.com/corpnewt/OCConfigCompare) as a direct solution is coupled with additional infrastructure demands or rewrites that would fall outside the scope of the project (i.e. an intermediate parsed format or AST still requires knowledge of the version-specific defaults). ### ii. Supplemental config formats (i.e. config.yml) The solution proposed by this project is to introduce a supplemental config/diff format additionally with a set of tools to support new and existing configurations and workflows with as little friction as possible. The idea is to add a supplemental format that transparently reflects the made configuration changes. This is done by giving first-class treatment to declarative configuration formats that are much more easily handled by versioning tools like git. In the case of merging two existing configurations (aka an existing config.plist with a newer Sample.plist), merging and emitting the same declarative format that reflects the same changes. Resembling the tree view of ProperTree, a single-property change can instead be written with YAML as: ```yaml # Changes default macOS installer/keyboard language to English. # - You can find a list of supported languages below (convert HEX -> ASCI): # https://github.com/acidanthera/OpenCorePkg/blob/master/Utilities/AppleKeyboardLayouts/AppleKeyboardLayouts.txt NVRAM: Add: 7C436110-AB2A-4BBB-A880-FE41995C9F82: prev-lang:kbd: String | "en-US:0" # -- or -- Misc.Boot.PickerMode: String | "External" ``` Or to add a new boot arg to a file, *preserving* existing values: ```yaml NVRAM: Add: 4D1FDA02-38C7-4A6A-9CC6-4BCCA8B30102: @append(",") revpatch: String | "sbvmm" 7C436110-AB2A-4BBB-A880-FE41995C9F82: @append(" ") boot-args: String | "foo" ``` ### iii. Build configuration formats (i.e. build.yml) Building off of the same ideas presented with handling config.plist files are with binary formats (i.e. SSDTs, Kexts, Drivers, etc). However, this introduces another issue where maintaining an existing project (or creating a new one) requires sourcing and re-downloading Kexts. Newer builds of Kexts may introduce breaking changes or a behavioral bug that reinforces the same need for backtracking, or lead to user-error where a configuration has unintentionally changed. It is also often desirable to include decompiled `.dsl` files to comment on and document SSDT code. In case of a regression, having the source code made available helps in giving transparency to code changes using tools like git. However, maintaining two copies of the compiled and decompiled SSDTs can be arduous and introduce merge conflicts where two binary formats don't agree with each other. The solution proposed by this project is a build configuration file format with an accompanying build system, extending upon the ideas presented with the `config.yml` format. This introduces additional composability by allowing users to specify desired versioning constraints and support incremental builds of existing projects. This also allows for injecting config.plist properties for ACPI/Kext/etc entries without coupling their configuration to the `config.yml` or config.plist file. Several examples can be shown using a fully integrated build environment (using a naive bash implementation of the ideas from this project): - [ASUS-ZenBook-Duo-14-UX481](https://github.com/Qonfused/ASUS-ZenBook-Duo-14-UX481-Hackintosh/tree/main/src) - [ASUS-ZenBook-Pro-Duo-15-UX581](https://github.com/Qonfused/ASUS-ZenBook-Pro-Duo-15-UX581-Hackintosh/tree/main/src) - [OSX-Hyper-V](https://github.com/Qonfused/OSX-Hyper-V/tree/main/src) A concise example of the composability of this build configuration format can be shown with the example build.yml file. This is written to outline a structure resembling the contents of the `EFI/OC` directory:Background
```yaml --- build: DEBUG # The OpenCore build type version: latest # The OpenCore version --- ACPI: SSDT-PNLF: "*" # Marks this SSDT as bundled with OpenCore Drivers: - AudioDxe - HfsPlus @ifdef(RELEASE) - OpenCanopy @endif - OpenRuntime - ResetNvramEntry - Rts5227S Kexts: WhateverGreen: latest # Fetches the latest build from the Dortania build repo AirportItlwm-Ventura: specifier: OpenIntelWireless/itlwm=^2.2.0 properties: MaxKernel: "22.99.99" MinKernel: "22.0.0" AirportItlwm-BigSur: specifier: OpenIntelWireless/itlwm=^2.1.0 properties: MaxKernel: "20.99.99" MinKernel: "20.0.0" RestrictEvents-1: "acidanthera/RestrictEvents#e5c52564f5bca1aebbd916f2753f5a58809703a8" RestrictEvents: specifier: acidanthera/RestrictEvents#foo # foo is used to demonstrate handling of conflicting properties branch: force-vmm-install properties: Comment: "Workaround for macOS Sonoma OTA updates on unsupported SMBIOSes" VoodooI2C: specifier: VoodooI2C/VoodooI2C=2.8 bundled: # This property is used to explicitly mark which plugins to be loaded - VoodooI2C/VoodooI2CServices - VoodooI2C/VoodooGPIO - VoodooI2C/VoodooInput Foo: "file:./Kexts/UTBMap.kext" Tools: - ControlMsrE2 - OpenShell ```docs/example/src/build.ymlContributing
Please refer to the project's CONTRIBUTING doc for instructions on how to set up the development environment and tools used by this project. These are used to ensure code quality and consistency across the project, and are subject to change as the project evolves.
If you want to use the library and CLI locally (without cloning the project), you can install the latest commit as a git dependency by running:
In the case of a poetry project, you can configure a VCS dependency with something like: