Closed leamas closed 2 years ago
Working with Tony Voss, to enable him to make his own builds using the workflow. His experiences will help to refine this.
OK. Perhaps late, but a hint: Use the sd3.0.0 or the latest beta rather than shipdriver/master, the tags are definitely more stable. Which is expected.
@rasbats:
Looking into https://opencpn-manuals.github.io/main/AlternativeWorkflow/InstallConfigure.html
The three first bullets are crystal clear to me: Configure OpenCPN/plugins, Cloudsmith and the builders. So far, so good.
However, the remaining two "Cloud Service Changes and "Github" are hard for me to wrap my head around. What are they about, really? If you would write sentence on the top of each page describing it, what would it be?
I'm probably just dumb. Need to see the light...
hm... Looking more into Cloud Service changes it seems that this is all about handling files with missing execute permissions after copying them in Windows.
But this does not happen when we use update-templates, it uses git to copy files so the permissions are OK.
Can we just drop this page? Or?
... and when I look into the Github page I see something which might be described as "Final checks and testing the build configuration" or so.
Is this correct?
two "Cloud Service Changes and "Github"
Removed.
@Rasbats Hm. After https://github.com/Rasbats/managed_plugins/pull/43 this looks in my eyes as a reasonably consistent structure, more on a step-by-step basis. But that's just me.
Thoughts? How does it look in your eyes?
I think you may have lost sight of converting legacy plugins. Tried using update-templates with a plugin recently and it kept going round in a loop and not adding any directories. The script assumes the prescence of directories, many of which do not exist in the legacy plugin.
Worked around this by copying a lot of directories from ShipDriver and then was able to run update-templates to check the layout. Because of your work with the scripts the directory copying process is straightforward and moving stuff from the original CMakeLists.txt into Plugin.cmake is easy. Search and replace takes care of most of the issues.
Is another page needed in the manual? Or an update to update-templates?
Maybe some reassurance for the new developer is needed. What started as Checklist.pdf has morphed! A lot of stuff to confuse the newbie, especially if they do not use git cli or bash scripts. Most of my processes use a GUI.
Having said all this the PIM is the easiest way to find and install plugins.
Another thought ... I am avoiding beta catalog plugin versions because no-one is going to find them in OpenCPN 5.6.0. Must add a note on the beta plugins manual page telling people how to use the beta catalog - CatalogExpert=1
.
Tried using update-templates with a plugin recently and it kept going round in a loop
Which plugin?
Is another page needed in the manual? Or an update to update-templates?
update-templates should be fixed, for sure.
Most of my processes use a GUI.
yes, and they fail. The copy using Windows gui tools just cannot handle permissions, and you need to add long and complicated docs how to fix it. Another problem is that it so damned hard to update long descriptions with screenshots when files and and directories are changed. Obvious in the existing docs before we simplified.
Another aspect is that not all devs uses Windows, and the gui solutions tends to become locked to the OS and it's tools.
In short: The only way I can think of which might become stable and platform independent is using the cli.
Maybe some reassurance for the new developer is needed.
There is always room for that. You are right, this has morphed from a simple checklist to a more full description. Still, a developer need to handle all these steps to actually adapt a plugin, we will not make anything easier by hiding stuff.
I note that you don't say the new structure is a mess worse than what we had.
Most of my processes use a GUI.
yes, and they fail. [...]
I'm missing the core point here: The gui processes cannot be automated with a reasonable effort. Which means more manual and error-prone work for the dev using a GUI.
Which plugin?
It was EarthExplorer. Try using the commit 273f652ac40abce6b6ac83da3a33d3708cde8570
This is before moving it to a managed version.
yes, and they fail. The copy using Windows gui tools just cannot handle permissions
Agree but a mix of gui and bash update-templates
works. Another instance is the use of the github.com page and Fetch upstream and other web pages to sort the auto branch of the OpenCPN/plugins fork.
Should have said ... I was using git bash to run update-templates.
It was EarthExplorer. Try using the commit 273f652...
hm... Using update-templates on that I see a problem: after bootstrapping I try _./updatetemplates -l to see available tags, but it fails due to missing shipdriver remote. Will fix, in an upcoming PR.
When I work around that by adding the shipdriver remote manually and runs _./updatetemplates sd3.0.1-beta1 I get the following history:
bbfbf3d (HEAD -> master) cmake: Update TemplateVersion to 8a44c67
f3b02a0 Templates: Removing obsoleted and unused files.
192c848 Templates: Updating other files
23df184 Templates: Updating libs
196062a Templates: Updating buildwin
3131691 Templates: Updating scripts
970b0d6 Templates: Updating ci
cf77576 Templates: Updating cmake
ca1bc3d update-templates: UPdated to latest version.
33bb5d4 Add update-templates script
273f652 (tag: v0.3-beta) v0.3-beta
This looks more or less like your log, besides this:
b0f99d ut
91a758f ut
81fef34 Move to managed version
99170ac Update update-templates
b09b8c3 Update update-templates
99075bb Update update-templates
adcc6d1 ut
3cd3f6b Update update-templates
142a699 Update update-templates
What was going on here? Did you have problems with update-templates' self-updating?
Should have said ... I was using git bash to run update-templates.
So did I.
Did you have problems with update-templates' self-updating?
Yes ... that was the loop that I did not get out of. Each time the script asked for a commit. Did that and used update-templates again, with the same result.
I was using sd3.0.0: ./update_templates sd3.0.0
and ./update_templates shipdriver/v3.0
hm... what you need to do there after the first run which updates the script is something like:
$ git add update-templates
$ git commit -m "update-templates: Updated to latest version"
Did you try something else here?
After using update-templates Github Desktop told me I had files to commit, which I did (using the gui!). Looking at the history I can see that update-templates was updated. But it failed to be used, apart from requesting another copy of update-templates.
Perhaps you should avoid the gui? It works for me, not using any gui. And it fails for you, using the gui. Conclusion?
If so, this would not be the first time the GUI failed us. It's hard for me to understand what is actual the problem here since gui handling and results are not that easy to communicate.
Digging a little. I updated update-templates to sd3.0.1 in ca1bc3d. I can see that this worked as anticipated:
$ git diff --stat sd3.0.1-beta1 ca1bc3d update-templates
$
That is, no diff. When I had updated update-templates it was as of sd3.0.1-beta1
Your first update update-templates commit in that log is 142a699. However, after that commit update-templates was neither as of sd3.0.1 nor shipdriver/v3.0. It's not as of shipdriver/master either:
$ git diff --stat shipdriver/v3.0 142a699 update-templates
update-templates | 89 +++++++++++++++++++++++++++--------------------------------------------------------------
1 file changed, 27 insertions(+), 62 deletions(-)
$ git diff --stat shipdriver/master 142a699 update-templates
update-templates | 118 +++++++++++++++++++++++++++++-------------------------------------------------------------
1 file changed, 38 insertions(+), 80 deletions(-)
$ git diff --stat sd3.0.1-beta1 142a699 update-templates
update-templates | 89 +++++++++++++++++++++++++++--------------------------------------------------------------
1 file changed, 27 insertions(+), 62 deletions(-)
Ergo: something in how you commit the changes in update-templates seems broken.
You do follow the bootstrap procedure in UPDATE_TEMPLATES.md as the very first step, right?
You do follow the bootstrap procedure in UPDATE_TEMPLATES.md as the very first step, right?
Erm ... No.
Been here before. Is it a permissions issue caused by copying the update-templates script from the ShipDriver clone?
I'm not convinced that the bootstrap is the culprit here. But it seems important that we do the same thing so we can sort out the actual difference between you and me i. e., why it works for me but not you.
Is it a permissions issue caused by copying the update-templates script from the ShipDriver clone
Probably not, you cannot have any execute permissions in windows anyway.
Rubbish weather. I will try to reproduce the problem with another of my beta plugins this morning.
oBabel_pi https://github.com/rasbats/obabel_pi Same issue with tag sd3.0.0. I will try with some other tags.
Mike@vostro-64 MINGW64 ~
$ cd /c/Users/Mike/Documents/GitHub/oBabel_pi
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$ bash update-templates
Usage:
update-templates [-T] <treeish>
update-templates [-h | -l]
Parameters:
treeish:
A shipdriver tag or branch.
Options:
-l List available shipdriver tags
-h Print this message and exit
-T Test mode, do not auto-update
Examples:
update-templates -l -- List available tags
update-templates sd3.0.0 -- Update from sd3.0.0 tag
update-templates shipdriver/v3.0 -- Update from v3.0 release branch
update-templates shipdriver/master -- Update from development branch
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$ bash update-templates sd3.0.0
Please commit or stash pending changes. Aborting.
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$ bash update-templates sd3.0.0
Adding new shipdriver remote
Fetching shipdriver
remote: Enumerating objects: 5314, done.
remote: Counting objects: 100% (348/348), done.
remote: Compressing objects: 100% (177/177), done.
remote: Total 5314 (delta 192), reused 274 (delta 171), pack-reused 4966
Receiving objects: 100% (5314/5314), 4.08 MiB | 1.98 MiB/s, done.
Resolving deltas: 100% (3176/3176), done.
From https://github.com/Rasbats/shipdriver_pi
* [new branch] 2.1.b4.leamas -> shipdriver/2.1.b4.leamas
* [new branch] 4.8.6 -> shipdriver/4.8.6
* [new branch] b4-drone.io -> shipdriver/b4-drone.io
* [new branch] master -> shipdriver/master
* [new branch] patch-1 -> shipdriver/patch-1
* [new branch] patches -> shipdriver/patches
* [new branch] pr/13 -> shipdriver/pr/13
* [new branch] revert-316-local-build -> shipdriver/revert-316-local-build
* [new branch] v3.0 -> shipdriver/v3.0
* [new tag] sd3.0.1-beta1 -> sd3.0.1-beta1
* [new tag] v0.6_beta -> v0.6_beta
* [new tag] sd3.0.0 -> sd3.0.0
* [new tag] sd3.0.0-beta1 -> sd3.0.0-beta1
* [new tag] sd3.0.0-beta2 -> sd3.0.0-beta2
* [new tag] sd3.0.0-rc1 -> sd3.0.0-rc1
* [new tag] v0.1 -> v0.1
* [new tag] v0.2 -> v0.2
* [new tag] v0.3 -> v0.3
* [new tag] v0.4 -> v0.4
* [new tag] v0.5 -> v0.5
* [new tag] v2.0 -> v2.0
* [new tag] v2.1 -> v2.1
* [new tag] v2.2 -> v2.2
* [new tag] v2.3 -> v2.3
* [new tag] v2.4 -> v2.4
* [new tag] v2.6.2.0 -> v2.6.2.0
* [new tag] v2.7.0.0-beta -> v2.7.0.0-beta
* [new tag] v2.7.1.0 -> v2.7.1.0
* [new tag] v2.8.0.0-beta -> v2.8.0.0-beta
* [new tag] v2.8.1.0-beta -> v2.8.1.0-beta
* [new tag] v2.8.2.0-beta -> v2.8.2.0-beta
* [new tag] v2.8.3.0-beta -> v2.8.3.0-beta
* [new tag] v2.8.4.0-beta -> v2.8.4.0-beta
* [new tag] v2.8.4.1-beta -> v2.8.4.1-beta
* [new tag] v2.8.4.2-beta -> v2.8.4.2-beta
* [new tag] v2.8.4.3-beta -> v2.8.4.3-beta
* [new tag] v2.8.4.4-beta -> v2.8.4.4-beta
* [new tag] v2.8.4.5-beta -> v2.8.4.5-beta
* [new tag] v2.8.4.6-beta -> v2.8.4.6-beta
* [new tag] v2.8.4.7-beta -> v2.8.4.7-beta
* [new tag] v2.8.5.0-beta -> v2.8.5.0-beta
* [new tag] v2.8.5.1-beta -> v2.8.5.1-beta
* [new tag] v2.8.6.0 -> v2.8.6.0
* [new tag] v2.8.7.0 -> v2.8.7.0
* [new tag] v2.8.7.2 -> v2.8.7.2
Checking for updates of updates-templates script
Updated 1 path from ced7eb4
update-templates script is updated to latest version. Please commit
changes and re-run script.
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$ bash update-templates sd3.0.0
Removing existing shipdriver remote.
Adding new shipdriver remote
Fetching shipdriver
From https://github.com/Rasbats/shipdriver_pi
* [new branch] 2.1.b4.leamas -> shipdriver/2.1.b4.leamas
* [new branch] 4.8.6 -> shipdriver/4.8.6
* [new branch] b4-drone.io -> shipdriver/b4-drone.io
* [new branch] master -> shipdriver/master
* [new branch] patch-1 -> shipdriver/patch-1
* [new branch] patches -> shipdriver/patches
* [new branch] pr/13 -> shipdriver/pr/13
* [new branch] revert-316-local-build -> shipdriver/revert-316-local-build
* [new branch] v3.0 -> shipdriver/v3.0
Branch 'update.42b36fc' set up to track remote branch 'master' from 'shipdriver'.
Created temporary work branch update.42b36fc
Checking for updates of updates-templates script
Updated 1 path from f93c84b
update-templates script is updated to latest version. Please commit
changes and re-run script.
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$ bash update-templates sd3.0.0
Removing existing shipdriver remote.
Adding new shipdriver remote
Fetching shipdriver
From https://github.com/Rasbats/shipdriver_pi
* [new branch] 2.1.b4.leamas -> shipdriver/2.1.b4.leamas
* [new branch] 4.8.6 -> shipdriver/4.8.6
* [new branch] b4-drone.io -> shipdriver/b4-drone.io
* [new branch] master -> shipdriver/master
* [new branch] patch-1 -> shipdriver/patch-1
* [new branch] patches -> shipdriver/patches
* [new branch] pr/13 -> shipdriver/pr/13
* [new branch] revert-316-local-build -> shipdriver/revert-316-local-build
* [new branch] v3.0 -> shipdriver/v3.0
Checking for updates of updates-templates script
Updated 1 path from ced7eb4
update-templates script is updated to latest version. Please commit
changes and re-run script.
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$ git commit -m "New update-templates #3"
[master b0d79d2] New update-templates #3
1 file changed, 38 insertions(+), 80 deletions(-)
mode change 100644 => 100755 update-templates
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$ git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 822 bytes | 102.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (2/2), completed with 2 local objects.
To https://github.com/Rasbats/oBabel_pi.git
7a3100d..b0d79d2 master -> master
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$ bash update-templates sd3.0.0
Removing existing shipdriver remote.
Adding new shipdriver remote
Fetching shipdriver
From https://github.com/Rasbats/shipdriver_pi
* [new branch] 2.1.b4.leamas -> shipdriver/2.1.b4.leamas
* [new branch] 4.8.6 -> shipdriver/4.8.6
* [new branch] b4-drone.io -> shipdriver/b4-drone.io
* [new branch] master -> shipdriver/master
* [new branch] patch-1 -> shipdriver/patch-1
* [new branch] patches -> shipdriver/patches
* [new branch] pr/13 -> shipdriver/pr/13
* [new branch] revert-316-local-build -> shipdriver/revert-316-local-build
* [new branch] v3.0 -> shipdriver/v3.0
Deleted branch update.42b36fc (was 42b36fc).
Branch 'update.42b36fc' set up to track remote branch 'master' from 'shipdriver'.
Created temporary work branch update.42b36fc
Checking for updates of updates-templates script
Updated 1 path from f93c84b
update-templates script is updated to latest version. Please commit
changes and re-run script.
Mike@vostro-64 MINGW64 ~/Documents/GitHub/oBabel_pi (master)
$
bash update-templates shipdriver/master
works. Reverting and trying others.
This plugin was already in a managed version. Trying a non managed legacy plugin.
Please ~try~ use sd3.0.1-beta1, this is what I did
otidalroute_pi https://github.com/Rasbats/otidalroute_pi/
Mike@vostro-64 MINGW64 ~/Documents/GitHub/otidalroute_pi (master)
$ bash update-templates sd3.0.1-beta1
Removing existing shipdriver remote.
Adding new shipdriver remote
Fetching shipdriver
From https://github.com/Rasbats/shipdriver_pi
* [new branch] 2.1.b4.leamas -> shipdriver/2.1.b4.leamas
* [new branch] 4.8.6 -> shipdriver/4.8.6
* [new branch] b4-drone.io -> shipdriver/b4-drone.io
* [new branch] master -> shipdriver/master
* [new branch] patch-1 -> shipdriver/patch-1
* [new branch] patches -> shipdriver/patches
* [new branch] pr/13 -> shipdriver/pr/13
* [new branch] revert-316-local-build -> shipdriver/revert-316-local-build
* [new branch] v3.0 -> shipdriver/v3.0
Checking for updates of updates-templates script
Updated 1 path from 9509c49
update-templates script is updated to latest version. Please commit
changes and re-run script.
Mike@vostro-64 MINGW64 ~/Documents/GitHub/otidalroute_pi (master)
$ git commit -m "Update update-templates"
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
Mike@vostro-64 MINGW64 ~/Documents/GitHub/otidalroute_pi (master)
$ bash update-templates sd3.0.1-beta1
Removing existing shipdriver remote.
Adding new shipdriver remote
Fetching shipdriver
From https://github.com/Rasbats/shipdriver_pi
* [new branch] 2.1.b4.leamas -> shipdriver/2.1.b4.leamas
* [new branch] 4.8.6 -> shipdriver/4.8.6
* [new branch] b4-drone.io -> shipdriver/b4-drone.io
* [new branch] master -> shipdriver/master
* [new branch] patch-1 -> shipdriver/patch-1
* [new branch] patches -> shipdriver/patches
* [new branch] pr/13 -> shipdriver/pr/13
* [new branch] revert-316-local-build -> shipdriver/revert-316-local-build
* [new branch] v3.0 -> shipdriver/v3.0
Checking for updates of updates-templates script
Updated 1 path from 9509c49
update-templates script is updated to latest version. Please commit
changes and re-run script.
Mike@vostro-64 MINGW64 ~/Documents/GitHub/otidalroute_pi (master)
$
The script seems to fetch all the branches but I am still in the loop. Trying to find commit 9509c49.
Reason for nothing to commit
was that Desktop was used to make the commit and this was just a check.
bash update-templates shipdriver/v3.0
did not work.
bash update-templates shipdriver/master
did work.
Could not track down commit 9509c49.
let's focus on sd3.0.1-beta1, this is the release candidate. In particular, sd3.0.0 is too old.
Here is my log on otidalroute (git-bash). Note the 567-568 workaround to make update-templates -l work
561 git checkout 404f7ee
562 repo=https://raw.githubusercontent.com/Rasbats/shipdriver_pi/master
563 curl $repo/update-templates --output update-templates
564 git add update-templates
565 git commit -m "Add update-templates script"
566 ./update-templates -l
567 grep 'remote add' update-templates
568 git remote add shipdriver https://github.com/Rasbats/shipdriver_pi.git
569 ./update-templates -l
570 ./update-templates sd3.0.1-beta1
571 git add update-templates
572 git commit -m "update-templates: Update to sd3.0.1-beta1"
573 ./update-templates sd3.0.1-beta1
The git log is then
a5056a7 (HEAD) cmake: Update TemplateVersion to 8a44c67
d5b34e3 Templates: Removing obsoleted and unused files.
3a50632 Templates: Updating other files
09e3ce6 Templates: Updating libs
73694de Templates: Updating buildwin
ab8c1ee Templates: Updating scripts
b0f7bc5 Templates: Updating ci
46a3728 Templates: Updating cmake
d6171f0 update-templates: Update to sd3.0.1-beta1
d4bed0b Add update-templates script
404f7ee ==
All looks good to me. If we could agree that sd3.0.1-beta1 works as expected we could move on, I guess...
BTW:
Checking for updates of updates-templates script
Updated 1 path from 9509c49
update-templates script is updated to latest version. Please commit
changes and re-run script.
Mike@vostro-64 MINGW64 ~/Documents/GitHub/otidalroute_pi (master)
$ git commit -m "Update update-templates"
On branch master
Your branch is up to date with 'origin/master'.
nothing to commit, working tree clean
Did you forget to add update-templates before committing here? Or was it also here committed elsewhere in the GUI?
Reason for nothing to commit was that Desktop was used to make the commit and this was just a check.
OK. That said, you should be able to repeat my history above and get the same results. I suggest that you leave the GUI out of the loop at first so we get some common ground.
Followed your log. Updated. Not yet pushed.
e09537b6872ada3f95ee59739f269ec9160475ff (HEAD -> master) cmake: Update TemplateVersion to 8a44c67
a06c5553557208d2e295a522bd490fab0ecefb25 Templates: Removing obsoleted and unused files.
b16035206a7fcc52a968acd7096774b33801b308 Templates: Updating other files
5eb4866b368156fab0c8fd975e078d9be9e1d6ad Templates: Updating libs
f9968f138b206e30b2f42df47d484dd2d2317081 Templates: Updating buildwin
eb59bc7a323a600042a3562a92ff52dcbbb1843b Templates: Updating scripts
60c9657c1b7cdd23128c2806161a2aaa09ac2186 Templates: Updating ci
46f82f2b411c76031ed9caeb4f679d132503067b Templates: Updating cmake
c59b19f1e043eb8a03b139d8297e3fc8c972bdf7 update-templates: Update to sd3.0.1-beta1
fe9d1b7a54d1b6d2d8921fe635987ae76970ba87 (origin/master, origin/HEAD) Add update-templates script
To me, everything looks OK.
If you followed my log your current state is detached. If you want to push it you need to do something like:
$ git branch -m master master.old
$ git checkout -b master
$ git push --set-upstream origin master:master
However, the interesting question is why this works while your previous attempts didn't. Have you any idea?
Have you any idea?
Previously branch v3.0 was used to supply update-templates.
v3.0 has the tag sd3.0.1-beta1 but the script does not find it?
With this:
repo=https://raw.githubusercontent.com/Rasbats/shipdriver_pi/master
curl $repo/update-templates --output update-templates
we are getting update-templates from the master branch. Has v3.0 been updated with the master version of update-templates? Is there a difference?
So my mistake is not bootstrapping. Could update-templates handle this error?
So my mistake is not bootstrapping.
Ouch...
Could update-templates handle this error?
Probably not.
After all, "handling" would mean a change to update-templates, a change which won't be available unless the bootstrap procedure is followed. It's basically a chicken-egg problem.
But I will sleep on it, perhaps there is a clever solution
The simple fix for now is probably to warn about copying stuff instead of following the bootstrap procedure in the manual. I'll make a PR, I have another simple fix pending.
A bit confusing (for me) ... we get update-templates from the master branch of SD to use the v3.0 branch of SD for the update?
Bootstrapping is confusing. I think we just need to use what works.
It's a little bit on shaky ground since master by definition is unstable. We would be better off using a tag here, but the first usable tag is likely sd3.0.1 which yet doesn't exist.
BTW: the normal usecase is a tag, not a branch.
Closing this. We have better structure in place. Not to say it's perfect...
Contemplating Rick's experiences I think that there is an implicit flow in the docs on how to adapt a plugin to SD templates
However, this implied overall flow is not clearly described anywhere. Also, in particular the important and large CI setup would benefit from a more structured and numbered step-by-step structure.
In short: the manual would benefit from some refactoring :)