search

jilleb / mib2-toolbox

The ultimate MIB2-HIGH toolbox.
MIT License
625 stars 145 forks source link

Use metainfo2.txt concatenation installation mechanism. #125

Closed andrewleech closed 3 years ago

andrewleech commented 3 years ago

This PR replaces the PersonPOI based installer with one that works by adding the installation details after the signature block in the install script from an official VW map pack. A flaw in the SWDL system allows this method of bypassing the signature requirements to install firmware.

Based on https://github.com/jilleb/mib2-toolbox/issues/1#issuecomment-763540383 and https://github.com/jilleb/mib2-toolbox/issues/122

This PR also uses this to run a final script during the install to automate the script installation, meaning users will no longer need to manually run the GEM command on first use to copy the scripts.

Full description and documentation updates TBD

This new install mechanism also triggers a software update SVM error code which requires a VAG / VCDS / OBDeleven XOR process to clear, though it may be possible to write a script to do this in future.

jilleb commented 3 years ago

Great work! I'll review and merg later today!

andrewleech commented 3 years ago

I'd hold of merging until it's had a bit more testing - I haven't tested the commit I just pushed with the updated script, the instructions need updating etc, confirm there's no bad side effects to using the DVD block etc.

I've also got the version set to 1 which means it'll need the SWDL download enabled and then manually force it to install, which will need instructions.

Alternatively we can set it to a higher number that'll likely auto-install correctly but users will need to turn manually force the DVD install if/when then install a firmware pack in future.

jilleb commented 3 years ago

Allright. I'll put it to a experimental branch then 😎

olli991 commented 3 years ago

I'd hold of merging until it's had a bit more testing - I haven't tested the commit I just pushed with the updated script, the instructions need updating etc, confirm there's no bad side effects to using the DVD block etc.

I've also got the version set to 1 which means it'll need the SWDL download enabled and then manually force it to install, which will need instructions.

Alternatively we can set it to a higher number that'll likely auto-install correctly but users will need to turn manually force the DVD install if/when then install a firmware pack in future.

I think we can fix this with my method from the RadioStationDB install. I defined an extra component. This way it will set another component in the update which isn't there in normal updates. So no SWDL needed.

Needs to add something like:

. . .
ConfigComponent = "DVD"
PacketName = "GreenEngineeringMenu"
ConfigCheckPath = "-"
ConfigFinalizePath = "/net/mmx/mnt/app/eso/hmi/engdefs"
IncludeModules = ",ESD,"

Will test myself.

olli991 commented 3 years ago

Okay nevermind. DVD seems to work another way. you don't need to config component. Like you already done with path Toolbox this defines a new module. Somehow every module will list with current version 2. I don't know why but this shouldn't be a problem. If we set Toolbox as path like you done in your file we have a new module entry which only correspond to our toolbox installations.

Further investigations:

olli991 commented 3 years ago

Hmm final script isn't running for me. Gives me errorcode 113. Same es in my test before.

andrewleech commented 3 years ago

Ok, I've set the install version to 3500 so it should now install without any manual software download stuff, it should basically just work.

If this does end up causing conflicts with official fw update we can either write instructions for how to manually install the official fw, or yeah as suggested add extra stuff to the final script to "undo" any DeviceRelease entry stored.

andrewleech commented 3 years ago

Hmm final script isn't running for me. Gives me errorcode 113. Same es in my test before.

Darn, could you try from my first commit on this branch? I might have got the updated sha1 hashes wrong or something - I'm planning on writing a script to automate this.

olli991 commented 3 years ago

Ok, I've set the install version to 3500 so it should now install without any manual software download stuff, it should basically just work.

If this does end up causing conflicts with official fw update we can either write instructions for how to manually install the official fw, or yeah as suggested add extra stuff to the final script to "undo" any DeviceRelease entry stored.

We don't need to include it. So no number will be set. completly delete this version info ;-)

andrewleech commented 3 years ago

Ok, I've set the install version to 3500 so it should now install without any manual software download stuff, it should basically just work. If this does end up causing conflicts with official fw update we can either write instructions for how to manually install the official fw, or yeah as suggested add extra stuff to the final script to "undo" any DeviceRelease entry stored.

We don't need to include it. So no number will be set. completly delete this version info ;-)

Oh, if that works and installs by default that's even better! I'll try that next time I get some dev time in my car, as well as verifying the hashes.

olli991 commented 3 years ago

Hmm I made some adjustments to the script which errors in 113. Now i went back to last commit and i get error 119 now in Unit common_toolsDir will try again first version again now.

Maybe get in touch with me via telegram, would unclutter the threads i think :D t.me/olli991

Edit: Even first commit version errors in 119 for me.

olli991 commented 3 years ago

I opend up a experimental test branch for the new installer: https://github.com/jilleb/mib2-toolbox/tree/experimental-new-intall I thought this would be much easier to compare everything without merging with master.

Current state:

andrewleech commented 3 years ago

@olli991 Could you check if this: USB Drive.zip works for you?

olli991 commented 3 years ago

It actually went trough without errors. Only thing i noticed is i have a NOK for RadioStationDB in the final Log. wtf? :D

i will investigate yours and mine version... so finalscript worked... should be possible to get this running even with my more edited version.

jilleb commented 3 years ago

It actually went trough without errors. Only thing i noticed is i have a NOK for RadioStationDB in the final Log. wtf? :D

i will investigate yours and mine version... so finalscript worked... should be possible to get this running even with my more edited version.

The NOK could also happen when it didn't update because the version on the unit was identical to the version that was already there.

olli991 commented 3 years ago

There is absolutly no RadioStationDB Update even mentioned in the metainfo or shown via custom swdl before. that bothers me.

olli991 commented 3 years ago

Okay i found a possible flaw you finalscript has a blank row at the end mine not. Tried it, didn't work. Checked the files with winmerge.... somhow mine is completly yellow even though content is literally the same... somting broken with my file i think... i now copied you scriptfile and edited to my commit.... let's see

olli991 commented 3 years ago

My files were fucking coded in Windows (CR+LF)... they need to be in Unix (LF). fuck me...

Now everything works. The door is open we can perform everthing with modified update! 🎉

olli991 commented 3 years ago

@andrewleech did you already tested if we could also save the finalscript somwhere else? Will test it tomorrow if i can. This way we could make the structure even more simplified.

We also need to know if mapcare can block this update.

A now working and restructured build can be found in the experimental branch. Would be good if we investigare more on this build, i think.

andrewleech commented 3 years ago

Ah wow, I hate line endings so so much. Great you found the difference! This is a crucial finding too - as a default install of git on windows will download all text files with the line endings modified this way. We can/should add a .gitattributes file to the repo which tells git not to do this.

I haven't tried changing the dir of final script, but I'm sure it can be changed a bit. It'll just need to be somewhere with a hashes file, but it's probably easiest to keep it in its own folder so as to not need to update hashes too often.

andrewleech commented 3 years ago

Yeah it'll be interesting if the mapcare swap timeout blocks this, I kind of doubt it seeing as the navdb items are now blocked out from the installer. If that does cause a problem we should just be able to switch to a firmware update as the donor file.

olli991 commented 3 years ago

I prepared metainfo but didn't test so far. Don't know if this could work

#FinalScript = "./common/tools/0/default/finalScript.sh"
FinalScript = "./Install/final/finalScript.sh"
FinalScriptChecksum = "4928fe9cd3b741104e2a85c47f4d16fc9ff16d76"
FinalScriptMaxTime = "30"
FinalScriptName = "Final Script"

[Install\final\dir]
FileSize = "797"
CheckSum = "d995d9b1383b96e4e26380eb35b0af7487c4457e"

#[common\tools\0\default\dir]
#FileSize = "797"
#CheckSum = "d995d9b1383b96e4e26380eb35b0af7487c4457e"
andrewleech commented 3 years ago

So it turns out we don't need to reuse an existing config, I renamed my DVD block to MQB and it works perfectly.

I've got the folders cleaned up as suggested, works great.

I've added the gitattributes to enforce line endings, only to discover that metainfo2.txt does have to be CRLF, while the rest should be LF. The metainfo can't be parsed by the unit at all if its LF.

I still need to push up a fix for that last line ending need, but the rest of the branch is working perfectly. It shows the toolbox install, automatically installs fine without the manual download stuff, looks slick.

I've got a new video of the install process I'll publish too.

olli991 commented 3 years ago

I prepared metainfo but didn't test so far. Don't know if this could work

#FinalScript = "./common/tools/0/default/finalScript.sh"
FinalScript = "./Install/final/finalScript.sh"
FinalScriptChecksum = "4928fe9cd3b741104e2a85c47f4d16fc9ff16d76"
FinalScriptMaxTime = "30"
FinalScriptName = "Final Script"

[Install\final\dir]
FileSize = "797"
CheckSum = "d995d9b1383b96e4e26380eb35b0af7487c4457e"

#[common\tools\0\default\dir]
#FileSize = "797"
#CheckSum = "d995d9b1383b96e4e26380eb35b0af7487c4457e"

Nice so this idea with a completely new block actually worked. I suggest FinalScript works also in your commit from different source now. That's a really clean solution now.

Final steps:

andrewleech commented 3 years ago

Thanks for the py script review, I'll fix that!

I noticed your branch had other changes, but as I didn't know what they were for I wanted to keep my git commits/authorship with just the changes I've worked on for the installer.

If your other changes are to support the new installation method/paths it makes sense to merge them in, but other kinds of improvements should probably be in their own commits/branch?

On a positive note, I'm pretty sure I've got the line endings working correctly now. I downloaded the repo as a zip from my branch (https://github.com/andrewleech/mib2-toolbox/archive/new_installer.zip) and loaded it on a clean SD and it installed correctly.

Video is just uploaded: https://youtu.be/-6A-4-3qR_0

olli991 commented 3 years ago

I noticed your branch had other changes, but as I didn't know what they were for I wanted to keep my git commits/authorship with just the changes I've worked on for the installer.

I see. The other changes are added scripts, so improvments and additions to the toolbox.

Some changes are actually relevant to the update process i.e. the update_toolbox.sh. You're also missing the changelog of the esd.

I will manually copy your changes to the test branch then, if this is ok for you. But I'm not sure if we ever push this pull request then 🤣

andrewleech commented 3 years ago

@olli991 I've just merged / cherry picked the rest of your script fixes onto my branch, should be looking better now!

andrewleech commented 3 years ago

I'll take a look at the Readme later. I'm open to suggestions whether you think it's worth linking in the video, not sure how much help it is? Might be worth extending it with some details about how to enable developer / GEM in the first place, as well as just what to download / copy onto SD (this wasn't obvious to me when I started here).

olli991 commented 3 years ago

Can we somehow change this pullreqest to merge into the experimental branch? So this way we can collaborate in the branch and can pull this into the master?! I'm just getting into the mechanics of git tbh.

andrewleech commented 3 years ago

The experimental branch looked like it had a mixture / double up of commits, things weren't all that cleanly separated and most of my initial changes were kind of merged into your first commit (so not in my name anymore). I generally work on keeping my branches as clean as possible. I think I can give you access to my branch though if that helps?

edit: if you've got maintainer privs here you should already be able to push to my branch?

olli991 commented 3 years ago

ah ok... then I fucked it up, sorry. I just uploaded a local directory into a new branch. Local files where everything we've done so far. This way commits are gone. I can't fix this anymore I think.

olli991 commented 3 years ago

For documentation. We merged this pull request and my findings into the experimental branch. Further pullreqest on the new version will go into experimental till we merge experimental into master.

Further investigations:

Only thing left to do is readme. And maybe a new screenshot from the grennmenu itself for the readme. Also some housekeeping could be done.

andrewleech commented 3 years ago

@olli991 shame shortening the path didn't work, there must be some enforcement to the format of it. I did think last night, we could just rename our new Install folder to MQB then it'd align with metainfo and clean up the paths

jilleb commented 3 years ago

Indeed that would be right.

olli991 commented 3 years ago

I think its much clearer if we have a install folder tbh. I would leave it like this. Only thing we could think about is changing MQB to to somthing better. Because MQB stands for Modularer Quer Baukasten and thats literally a bit wide :D

andrewleech commented 3 years ago

Could change MQB to Toolbox and also call the folder that - I think that'd be as good as Install myself.

olli991 commented 3 years ago

Could change MQB to Toolbox and also call the folder that - I think that'd be as good as Install myself.

metainfo will look like this than [Toolbox\Toolbox\0\default\File]. But we have to live with it i guess

jilleb commented 3 years ago

It's not a big deal. On my original toolbox I didn't really think about nice folder names and such 😅😅 just went with what worked.

olli991 commented 3 years ago

It's not a big deal. On my original toolbox I didn't really think about nice folder names and such 😅😅 just went with what worked.

I have a fable for also good looking and / or intuitiv stuff. It's more work right, but could make it much easier to understand for newbies😬

jilleb commented 3 years ago

Yeah, I have to agree on that. Having clear naming decreases the amount of support needed 😅

andrewleech commented 3 years ago

Maybe change MQB\Toolbox to Toolbox\GEM?

andrewleech commented 3 years ago

@olli991 I've re-targeting this PR to the experimental branch, that worked now.

I've added a commit with the renaming Install -> Toolbox and updated scripts to suit. I also found a bunch of path issues/discrepancies in the scripts as I went through them for this, so there's a few cleanups there.

I've also updated the readme, let me know what you all think!

I can chuck this commit straight onto the experimental branch if you're happy with it too.

andrewleech commented 3 years ago

One more fix needed - the esd file needs it's line endings fixed too, it's currently broken / hash doesn't match

olli991 commented 3 years ago

One more fix needed - the esd file needs it's line endings fixed too, it's currently broken / hash doesn't match

How could this happen 🤔

andrewleech commented 3 years ago

One more fix needed - the esd file needs it's line endings fixed too, it's currently broken / hash doesn't match

How could this happen 🤔

The esd file isn't listed in gitattributes yet, so when I read my update hash script or was running on local LF version, but a clean zip from GitHub had it as CRLF (or the other way around, don't remember for sure)

olli991 commented 3 years ago

My local .esds are all crlf... maybe it doesn't matter?

andrewleech commented 3 years ago

It doesn't matter if it's LF or CRLF as long as the hash matches - changing the line endings changes the hash.

andrewleech commented 3 years ago

This final set of changes still need a live test. I'll try to do that tomorrow.

olli991 commented 3 years ago

This final set of changes still need a live test. I'll try to do that tomorrow.

Can do also later. We'll see. I'll give it a run if i can this evening. If it works I'll merge.

enricogialletta commented 3 years ago

Ottimo lavoro! Rivedrò e fonderò più tardi oggi!

Ciao jilleb ti scrivo per dirti che ho provato ad installare sul mio discover mib2 pro Volkswagen il tuo Mib2-toolbox ma quando vado ad installarlo l'autoradio mi dice che non ha potuto leggere il file, sulla SD ho messo tutto come da guida ho attivato opzione sviluppatore sullautoradio ma niente.