feat: Allow UUP Dump JSON API Url to be configured

[chadlwilson]

As noted in #27 uupdump.net has been down for many weeks. It may not come back. To allow more savvy/enthusiastic folks to use CrystalFetch with an alternate site, or a local instance - this change makes the UUP Dump URL user-configurable via Settings...

I'm not a SwiftUI expert, so not sure of the best way to validate the URL on input and such, but this PR

• Improves reporting of errors if there are errors during requests to include the base server called.
• logs response parsing errors to NSLog to assist with debugging
• Validated against a local install of UUP Dump per https://github.com/TuringSoftware/CrystalFetch/issues/27#issuecomment-1750303095

[osy]

I'm thinking of removing UUP support because it doesn't produce working ARM64 images and the new method does (and does not depend on a third party server). What's the argument for using UUP?

[Helloyunho]

I'm thinking of removing UUP support because it doesn't produce working ARM64 images and the new method does (and does not depend on a third party server). What's the argument for using UUP?

is there another method to download the image? I thought only UUP was the way to download...

[osy]

I'm thinking of removing UUP support because it doesn't produce working ARM64 images and the new method does (and does not depend on a third party server). What's the argument for using UUP?

is there another method to download the image? I thought only UUP was the way to download...

Have you tried opening the app?

[Helloyunho]

I'm thinking of removing UUP support because it doesn't produce working ARM64 images and the new method does (and does not depend on a third party server). What's the argument for using UUP?

is there another method to download the image? I thought only UUP was the way to download...

Have you tried opening the app?

probably

tho this is what I get when I try to list through all builds

[osy]

and... now we're back to my question:

https://github.com/TuringSoftware/CrystalFetch/pull/32#issuecomment-1760706587

[Helloyunho]

and... now we're back to my question:

#32 (comment)

ah lmao got it seems like it uses uup for any other builds then

so my honest opinion: I think it's okay to remove it if it sometimes produces wrong image but it would be nice to keep it for downloading old builds

[osy]

Okay tbh I don't know why you added yourself as a reviewer, said "lgtm" on a pull request, and made this off-topic thread of conversation to a question I was asking the author when you haven't read the code, haven't tested the code, or even opened the app.

[Helloyunho]

Okay tbh I don't know why you added yourself as a reviewer, said "lgtm" on a pull request, and made this off-topic thread of conversation to a question I was asking the author when you haven't read the code, haven't tested the code, or even opened the app.

oh sorry didn't mean to do that 😢

[chadlwilson]

Hi @osy - I have successfully used the app with this fix to build a working arm64 image off latest canary build, with a local running UUPdump.

So not sure what you mean by it not producing working arm64 images, perhaps I misunderstand.

If there is a method which doesn't involve UUP that's much better. This was just a way to unblock myself and potentially allow an alternate if another uupdump mirror API pops up.

[osy]

@chadlwilson see https://github.com/TuringSoftware/CrystalFetch/issues/8

What build number and are you able to run any UWP apps?

When you open the app, it is in ESD mode which will generate working images. If you click "all builds" it falls back to UUP mode.

[chadlwilson]

Ahh, ok. It was initially 25905.1000 if I recall correctly.

I'm not super familiar with what is a UWP app and what is not. While I hadn't known the root cause, this is possibly why when I first started Windows Security/Defender and Microsoft Store weren't there or working? Had not figured out why, had just assumed it was some UTM or VM weirdness/constraints.

I subsequently forced windows into the Insider Program canary channel, upgraded to 25967.1010 internally (because the latest build wasn't actually on uupdump) which restored Microsoft Store and Security Hub and such - possibly as a side effect if I understand the problem in #8.

I don't think this makes CrystalFetch/UUP useless to me because it still allowed me to start with an arm64 canary image build without an existing Windows VM or machine, which has fixes I wanted (25xxx on canary) which I then could get back into a working state for all apps. Not ideal but I appreciate this tool compared to my previous approach to get something arm64 for UTM.

Totally fine to ignore this PR if there is a better way forward though.

[osy]

I think it's fine to keep UUP if there's a good reason to. Why did you go to the legacy mode instead of building an image from the default ESP mode?

[chadlwilson]

FWIW, the reason I wanted to start with a canary build to sanity check is that 259xx+ has fixes for running x64 Ruby emulated on Windows arm64. ( https://github.com/oneclick/rubyinstaller2/issues/308#issuecomment-1606518939 )

[chadlwilson]

I think it's fine to keep UUP if there's a good reason to. Why did you go to the legacy mode instead of building an image from the default ESP mode?

I'm not sure I did anything intentional here, I just went to "All builds..." to be able to start with a 259xx canary and that put me into UUP land to populate the list. I had earlier tried to switch over to a canary from an existing 22xxx build but couldn't get Windows to let do it, so I thought I'd try from a fresh canary build.

It may have been not required, as there are a few hacks still required to get around the "your device doesn't meet requirements for insider preview builds blah blah" which I think relates to CPU detection. Possibly the same forcing I applied above (registry hacks to the insider preview channel) can be applied to other builds.

I'll have another go starting with the default 22621.1702 build that comes up and getting that onto canary with what I now know.

[osy]

I believe in the latest build of UTM, it supports TPM emulation and Secure Boot and so a newly created Windows VM should be able to enroll in canary.

[osy]

Oh I guess you still need some regedit hacks: https://github.com/utmapp/UTM/issues/5477#issuecomment-1640428160

Anyways if needing to download canary builds is a requested feature, I think that instead of adding a hack (specifying JSON URL), it would be better to just integrate the UUP backend into CrystalFetch. AFAIK, it just needs to run some PHP scripts and a web server right? This is probably overkill but we can just embed PHP and a simple webserver, run it on a localhost port and fetch from there. Alternatively, we can just re-code the PHP scripts into Swift.

[chadlwilson]

It also seems to need the PHP server to be bootstrapped via download/sync of a whole lot of metadata which current seems to rely on being updated by the UUP team (which presumably is somehow scraped from MS): see https://github.com/TuringSoftware/CrystalFetch/issues/27#issuecomment-1750303095

I'm not sure of the wisdom of relying on either this code or the underlying data (packs, fileinfo) directly inside CrystalFetch. Unless you have a connect to the UUP folks it may be that the same reasons leading to uupdump.net going down will also affect the availability of the data it relies upon.

Disclaimer: I am no UUP expert, just used my curiosity and need to get it going as quickly as possible.

[chadlwilson]

As discussed above, will close this in favor of a longer term solution to potentially move away from UUPDump or replicate what it does here. Can be reopened if necessary.