Use idx or pidx instead of scraping the html

atsams-rs / atpacks-svd-harvester

Obtains recent SVDs from Microchip ATPACKs repository.

0 stars 1 forks source link

Use idx or pidx instead of scraping the html #3

Open xoriath opened 4 months ago

xoriath commented 4 months ago

Reading through this, it seems like it's scraping the html... Microchip runs a CMSIS compliant pack repo, so there are 2 indexes available

https://packs.download.microchip.com/Microchip.pidx (all DFPs used in uVision)
[https://packs.download.microchip.com/index.idx (.gz)](https://packs.download.microchip.com/index.idx (.gz)) (all DFPs and all releases

Specs on the file format is available at https://open-cmsis-pack.github.io/Open-CMSIS-Pack-Spec/main/html/packIndexFile.html

Suggest to use one of these instead of relying on the HTML not changing...

michalfita commented 4 months ago

I'm open to PRs, you know.

But honestly, that's good idea that will give me sleepless nights. Moreover, if I understand this model correctly it would turn this tool into universal one, not Microchip specific. I'd need to rename it to something like cmsispack-svd-harvester. I looked into options of XML parsing and this would require consideration as I don't want half cooked solutions. I'm afraid the first slot I have capacity to work on this is Christmas period, I still struggle to work out release automation of PACs as I don't have enough space on my drive to hold build of all of them. And this is all unsponsored work.

I'm surprised anyone is interested in this tool, I thought I made it for myself to crack SVDs update for my PAC releases. I just checked downloads and there was recent spike what I find very interesting. Thanks anyway.

michalfita commented 4 months ago

@xoriath OK, I invested some time in writing the code for this idea, which is great in principle... but pretty bad in practice. The XML returned by https://packs.download.microchip.com/Microchip.pidx is not complete list of all families we want to support, SAMRH for example aren't on it. Hence, we'd have incomplete set in SVDs in return.

https://packs.download.microchip.com/index.idx is huge, very complicated to parse, and still probably doesn't cover all packs we're interested in to cover all families of chips. It's huge atmel namespace without known schema for it - I'm not interested in processing hundreds of megabytes of data, just to find the right pack.

This is another example of great concept dying thanks to bad execution.