EDK II upstream meta issue

[vit9696]

Reference bugzillas and links:

Open:

• https://bugzilla.tianocore.org/show_bug.cgi?id=2297
• https://bugzilla.tianocore.org/show_bug.cgi?id=2298
• https://bugzilla.tianocore.org/show_bug.cgi?id=2299
• https://bugzilla.tianocore.org/show_bug.cgi?id=2445
• https://bugzilla.tianocore.org/show_bug.cgi?id=2460
• https://bugzilla.tianocore.org/show_bug.cgi?id=2510
• https://bugzilla.tianocore.org/show_bug.cgi?id=2643
• https://bugzilla.tianocore.org/show_bug.cgi?id=2763
• https://bugzilla.tianocore.org/show_bug.cgi?id=2764

Resolved:

• https://bugzilla.tianocore.org/show_bug.cgi?id=2293
• https://bugzilla.tianocore.org/show_bug.cgi?id=2294
• https://bugzilla.tianocore.org/show_bug.cgi?id=2295
• https://github.com/acidanthera/OpenCoreShell/blob/master/Patches/024536.diff assume merged
• https://bugzilla.tianocore.org/show_bug.cgi?id=2296
• https://github.com/acidanthera/OpenCoreShell/blob/master/Patches/apfs-handle-size.diff assume ignore
• https://github.com/acidanthera/OpenCoreShell/blob/master/Patches/debug-printing.diff assume ignore
• https://bugzilla.tianocore.org/show_bug.cgi?id=2300
• https://bugzilla.tianocore.org/show_bug.cgi?id=2301
• https://bugzilla.tianocore.org/show_bug.cgi?id=2302
• https://bugzilla.tianocore.org/show_bug.cgi?id=2397
• https://bugzilla.tianocore.org/show_bug.cgi?id=2446
• https://github.com/acidanthera/bugtracker/issues/524
• https://bugzilla.tianocore.org/show_bug.cgi?id=2642
• https://bugzilla.tianocore.org/show_bug.cgi?id=564
• ~~https://github.com/acidanthera/OpenCoreShell/blob/master/Patches/legacy-switch-to-text.diff use OcConsoleControlEntryModeLib〜

• https://bugzilla.tianocore.org/show_bug.cgi?id=2054

UDK support is now abandoned by Intel, and TianoCore is trying to transition to a three month release cycle, which will neither get security backporting, nor receive enough testing [1] [2].

While quite displeasing for us, putting aside the security question of the matter, there are several more issues with how we handle UDK now:

• Some of the patches we need were partially merged to upstream EDK II, not UDK:
  • OFFSET_OF causing UBSan warning [3]
  • Multiple ShellPkg-related patches [4]
  • String manipulation assertions [15]
• Some of the packages we would like to use are now reworked or removed in EDK II:
  • DataHub deprecation in the next EDK II release [5]
  • DuetPkg removal from EDK II [6], this is a separate issue
  • EmuVariableDxe merge with VariableDxe [7]
• Some of the functions we use are now available in EDK II:
  • Base64Decode implementation in EDK II [8]
  • CharToUpper implementation in EDK II [9]
  • OpenFileByDevicePath, which is now different from EDK II, however [10]

I believe we should do the following:

• Make a new edk2 repository in acidanthera (without the use of github forking, as it breaks code search and we do not want any branches but our own).
• Make an opencore branch based on edk2-stable201903 (to avoid confusion with upstream).
• Merge ShellPkg [4] patches as cleanly as possible. I do not mind them being not formatted for upstream submission though, as this is not always possible, and can be done in parallel/future.
• The branch should be protected, just as everything else in acidanthera, but we will force push upon upgrading EDK II on biyearly(?) basis.

For integration:

• Replace CI edk2 base sources [11][12][13][14]
• Cleanup affected repositories (e.g. OcSupportPkg) from the use of UDK legacy, i.e. drop UDK support.
• Cleanup OpenCoreShell repository from patches (I would consider removing it entirely, but that is a separate issue).
• Update OpenCore documentation regarding EmuVariableDxe becoming VariableDxe.

1. https://edk2.groups.io/g/devel/topic/31222094#39482
2. https://github.com/tianocore/tianocore.github.io/wiki/UDK
3. https://github.com/acidanthera/OcSupportPkg#ocguardlib
4. https://github.com/acidanthera/OpenCoreShell/tree/master/Patches
5. https://bugzilla.tianocore.org/show_bug.cgi?id=1318
6. https://bugzilla.tianocore.org/show_bug.cgi?id=1322
7. https://bugzilla.tianocore.org/show_bug.cgi?id=1323
8. https://github.com/acidanthera/OcSupportPkg/blob/da6ac94/Include/Library/OcMiscLib.h
9. https://github.com/acidanthera/OcSupportPkg/blob/29a60df/Library/OcStringLib/OcUnicodeLib.c#L40
10. https://github.com/acidanthera/OcSupportPkg/blob/1de86ce/Library/OcFileLib/OpenFileByDp.c
11. https://github.com/acidanthera/OpenCorePkg/blob/797932b/macbuild.tool#L141
12. https://github.com/acidanthera/AppleSupportPkg/blob/c456e3f/macbuild.tool
13. https://github.com/acidanthera/AptioFixPkg/blob/208987d/macbuild.tool#L130
14. https://github.com/acidanthera/OpenCoreShell/blob/fc7bc93/macbuild.tool#L124
15. https://github.com/acidanthera/OpenCorePkg/blob/319b8061/Platform/OpenCore/OpenCoreNvram.c#L118-L123

[vit9696]

Initial buildscript migration for OpenCorePkg, AptioFixPkg, AppleSupportPkg is done. From this moment we drop UDK2018 support and require the use of edk2-stable201905.

[vit9696]

Changed to normal, as currently OpenCoreShell works to some level, and we are pretty much migrated to edk2-stable. Having drivers build within AppleSupportPkg is still quite important, however.

[vit9696]

Moved to first post.

[al3xtjames]

The hypervisor bit was added recently: https://github.com/tianocore/edk2/commit/77228269e7c3fac8c949ffed5e59182c4c521e58