search

OpenCPN / plugins

Container Project for an Integrated Plugin Management Facility
17 stars 20 forks source link

OpenCPN/plugins Alpha Branch Meta-url feature no longer works #381

Closed rgleason closed 3 years ago

rgleason commented 3 years ago

I would like to test Jon Gough's android builds for Watchdog and have pushed Watchdog v2.4.30.9 xmls into my meta-url "rg-alpha" , however these are no longer seen or recognized when I select the alpha channel under Opencpn Options Plugins.

This feature is very useful for testing and should be restored. It was working until recently.

These files should be available https://github.com/rgleason/plugins/tree/rg-alpha/metadata

I note that Jon Gough's meta url has been removed on his remote plugins repository. See https://github.com/rgleason/plugins/tree/Alpha/metadata and the contents of his meta-url https://github.com/rgleason/plugins/blob/Alpha/metadata/meta-url-alpha-jg-plugins.xml

While Jon may not use this feature which does not require bdbcat to accept a PR before being able to test a download, I find it very useful for ubuntu in a virtualbox and android testing of plugins because it eliminates the hastle of moving files around through different OS and it allows us to use the PIM interface with no intervention by someone else.

See https://opencpn.org/wiki/dokuwiki/doku.php?id=opencpn:developer_manual:pi_installer_opencpn_plugins#create_a_meta-url_on_alpha_branch

rgleason commented 3 years ago

Dave,

Moved the meta-url metadata to a new local directory "meta-url" for reference Reduced the metadata files.

It appears that the file ocpn-plugins.xml is being checked by the same standard as the metadata files. I don't believe that is necessary and that file should be excluded.

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ git commit -am "weather_routing_pi 1.13.30.0 + ocpn-plugins.xml"
metadata/weather_routing_pi-1.13.30.0-darwin-x86_64-11.4-macos.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.0-flatpak-aarch64-20.08-flatpak-arm64.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.0-flatpak-x86_64-18.08-flatpak.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.0-flatpak-x86_64-20.08-flatpak.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.0-msvc-x86_64-10.0.14393-MSVC.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.0-ubuntu-x86_64-20.04-focal-gtk3.xml validates

0 urls checked, 0 errors
ocpn-plugins.xml:2: element plugins: Schemas validity error : Element 'plugins': No matching global declaration available for the validation root.
ocpn-plugins.xml fails to validate

ocpn-plugins.xml.docx

rgleason commented 3 years ago

I've been having trouble opening Options > Plugins because in opencpn.log Downloader.cpp was always pointing to my plugins/rg-alpha directory looking for ocpn-plugins.xml. I had removed rg-alpha because it was no longer needed. I searched for an opencpn.ini reference to this and found none. From rgleason/plugins/master I git checkout -b rg-alpha and pushed to my remote and that did not fix the problem. Then I renamed opencpn.ini and started O again, the problem still existed. The OPtions> PI window would not open. So I have decided to update opencpn master locally and rebuild.

bdbcat commented 3 years ago

Rick... I do not understand how you are running the XML check scripts while doing a simple local "git commit -am...." Can you explain?

rgleason commented 3 years ago

In Bash CLI I just run python tools/ocpn-metadata generate --userdir metadata --destfile ocpn-plugins.xml --force

which builds a new ocpn-plugins.xml EDIT: I do this command first before committing. What should I be doing?

On the other issue, my now broken OpenCPN 5.5.0 I have removed it entirely and fetched and rebuilt and installed and now the Options > Plugins are not hanging up. I guess something got corrupted.

After installing a new OpenCPN 5.5.0 when I load the Alpha manual I got o-chart, ocpn-draw and testplugin listed. When I tried to install each one of them, o-chart worked, but ocpn-draw and testplugin had path problems and were not found. I notice now that these 3 plugins have a heavy check mark in a box to the right.

Screenshot (300)

It would be very helpful if the first message for downloading which shows the exact path was also repeated in the second message Download not Found 404 message for troubleshooting purposes.

rgleason commented 3 years ago

O-charts seems to download, but Jon's don't, getting 404.

I think I should be seeing weather_routing from the meta-url at some point here if I can get the WR ocpn_plugins.xml pushed up to rgleason/plugins Alpha directory (without trying to merge this into OpenCPN/plugins Alpha).

Screenshot (301)

bdbcat commented 3 years ago

Rick... In the Alpha catalog, there is a meta-url spec for Jon's repo: 

    <name> Jon Gough Alpha ODraw Plugin meta-url </name>
    <version> 1.0 </version>
    <release> 0 </release>
    <summary> meta-url to Jon's Alpha Plugin ocpn_draw_pi </summary>
    <api-version>  </api-version>
    <open-source> yes </open-source>
    <author>  </author>
    <source>  </source>
    <description>  </description>
    <target>all</target>
    <target-version> </target-version>
    <target-arch>noarch</target-arch>
    <tarball-url>  </tarball-url>
    <meta-url> https://raw.githubusercontent.com/jongough/plugins/Alpha/ocpn-plugins.xml </meta-url>
  </plugin>

When a meta-url is specified, that url points to a catalog. That catalog must exist, and must contain valid urls itself. Entries in that catalog will REPLACE entries (with the same common name) that are in the basic catalog loaded from Alpha. In this case, it seems that Jon's ocpn-plugins.xml that is fetched by the meta-url spec (shown above) actually contains a link to a tarball on Cloudsmith that is no longer valid.

But why do you care? ODraw is Jon's plugin. No real reason to try and load it from Alpha catalog. It is obviously broken there. This is Jon's business.

What about your plugins?

bdbcat commented 3 years ago

More info: A meta-url specification is NOT TIED to a particular plugin. It is tied to a developer's github repo, generally.

bdbcat commented 3 years ago

"I think I should be seeing weather_routing from the meta-url at some point here if I can get the WR ocpn_plugins.xml pushed up to rgleason/plugins Alpha directory (without trying to merge this into OpenCPN/plugins Alpha)."

That is exactly the correct procedure. Should work, if you push ocpn-plugins.xml to your repo.

On the metadata check:

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ git commit -am "weather_routing_pi 1.13.30.0 + ocpn-plugins.xml"
metadata/weather_routing_pi-1.13.30.0-darwin-x86_64-11.4-macos.xml validates

I really do not understand how this is happening on your system. All works fine here. Why would a local git commit invoke the xml check procedures.? I cannot find anything in the Alpha repo that even contains the word "validates".

bdbcat commented 3 years ago

At BASH prompt, plugins directory, Alpha branch, please do:

$ grep -r "validates" *

rgleason commented 3 years ago 

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ grep -r "validates" *

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ grep -r "validates" *

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ dir
check.bat            download_xml_bash.sh  meta-url               ocpn-plugins.xsd
check-metadata-urls  icons                 ocpn-plugin.xsd        README.md
DEVELOPER-INFO.md    Makefile              ocpn-plugins.xml       tools
download_xml.sh      metadata              ocpn-plugins.xml.docx  validate_xml.sh

I do have sub-directories for Tools metadata and git. There is a file validate_xml.sh in the root

validate_xml.sh.docx

rgleason commented 3 years ago

I do not believe I should have to push my remote rgleason/plugins Alpha changes up to OpenCPN/plugins Alpha because the meta-url does that.

So then we would need some validation of the metadata files, and then when they all pass, perhaps automatic creation of the ocpn-plugins.xml if possible?

Should I try again?

My local gitHub/plugins/ directory has .github/workflows with CI_Merge.yml and CI_Pull.yml

CI_Merge.yml has

  # only deal with push events
  push:
    branches: [ master, Beta, Alpha, Experimental ]

Ah Ha, it automatically creates the new ocpn-plugin.xml (I don't need to make one.)

     pip install pathlib
    - name: create ocpn-plugins.xml
      run: |
        python tools/ocpn-metadata generate --userdir metadata --destfile ocpn-plugins.xml --force

However its format seems to fail for some reason.

bdbcat commented 3 years ago

Rick.... The .github/workflows scripts do not run locally, unless special provisions are made. So no factor here. Please try again, your simple "commit -am..." CLI command.

rgleason commented 3 years ago

Dave, I updated Alpha by adding new WR metadata and commited and pushed. Where do I change the way this branch acts with respect to the scripts? I don't believe I changed anything about this repos with regard to running scripts.

"Schemas validity error : Element 'plugins': No matching global declaration available for the validation root." Perhaps the ocpn-plugins.xml is wrong in some way?

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ git add ocpn-plugins.xml

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ git commit -am "new ocpn-plugins xml"
metadata/weather_routing_pi-1.13.30.1-darwin-x86_64-11.4-macos.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-debian-x86_64-10-buster.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-flatpak-aarch64-20.08-flatpak-arm64.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-flatpak-x86_64-18.08-flatpak.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-flatpak-x86_64-20.08-flatpak.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-mingw-x86_64-10-mingw.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-msvc-x86_64-10.0.14393-MSVC.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-raspbian-armhf-10.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-raspbian-armhf-9.13.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-16.04-xenial.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-18.04-bionic-gtk3.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-18.04-bionic.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-20.04-focal-gtk3.xml validates

0 urls checked, 0 errors
ocpn-plugins.xml:2: element plugins: Schemas validity error : Element 'plugins': No matching global declaration available for the validation root.
ocpn-plugins.xml fails to validate

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$
rgleason commented 3 years ago

I compared a metadata file with ocpn-plugin.xml and found that one difference was

<build-gtk><build-gtk/>   versis       <build-gtk/>

so I paired them all to be

<build-gtk><build-gtk/>

and tried to commit.

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ git commit -am "manual fix ocpn-plugins xml"
metadata/weather_routing_pi-1.13.30.1-darwin-x86_64-11.4-macos.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-debian-x86_64-10-buster.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-flatpak-aarch64-20.08-flatpak-arm64.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-flatpak-x86_64-18.08-flatpak.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-flatpak-x86_64-20.08-flatpak.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-mingw-x86_64-10-mingw.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-msvc-x86_64-10.0.14393-MSVC.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-raspbian-armhf-10.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-raspbian-armhf-9.13.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-16.04-xenial.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-18.04-bionic-gtk3.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-18.04-bionic.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-20.04-focal-gtk3.xml validates

0 urls checked, 0 errors
ocpn-plugins.xml:26: parser error : Opening and ending tag mismatch: build-gtk line 19 and plugin
  </plugin>
           ^
ocpn-plugins.xml:48: parser error : Opening and ending tag mismatch: build-gtk line 41 and plugin
  </plugin>
           ^
ocpn-plugins.xml:70: parser error : Opening and ending tag mismatch: build-gtk line 63 and plugin
  </plugin>
           ^
ocpn-plugins.xml:92: parser error : Opening and ending tag mismatch: build-gtk line 85 and plugin
  </plugin>
           ^
ocpn-plugins.xml:114: parser error : Opening and ending tag mismatch: build-gtk line 107 and plugin
  </plugin>
           ^
ocpn-plugins.xml:136: parser error : Opening and ending tag mismatch: build-gtk line 129 and plugin
  </plugin>
           ^
ocpn-plugins.xml:158: parser error : Opening and ending tag mismatch: build-gtk line 151 and plugin
  </plugin>
           ^
ocpn-plugins.xml:180: parser error : Opening and ending tag mismatch: build-gtk line 173 and plugin
  </plugin>
           ^
ocpn-plugins.xml:202: parser error : Opening and ending tag mismatch: build-gtk line 195 and plugin
  </plugin>
           ^
ocpn-plugins.xml:224: parser error : Opening and ending tag mismatch: build-gtk line 217 and plugin
  </plugin>
           ^
ocpn-plugins.xml:268: parser error : Opening and ending tag mismatch: build-gtk line 261 and plugin
  </plugin>
           ^
ocpn-plugins.xml:291: parser error : Opening and ending tag mismatch: plugin line 247 and plugins
</plugins>
          ^
ocpn-plugins.xml:292: parser error : Premature end of data in tag plugin line 203
rgleason commented 3 years ago

Have now removed all empty pairs like this

<build-gtk><build-gtk/>

Still get errors.

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-18.04-bionic.xml validates

0 urls checked, 0 errors
metadata/weather_routing_pi-1.13.30.1-ubuntu-x86_64-20.04-focal-gtk3.xml validates

0 urls checked, 0 errors
ocpn-plugins.xml:2: element plugins: Schemas validity error : Element 'plugins': No matching global declaration available for the validation root.
ocpn-plugins.xml fails to validate

How do we get the script to make a good ocpn-plugin.xml?

rgleason commented 3 years ago

Ok, I was incorrectly trying to build ocpn-plugins.xml DON'T do that! Just update the Alpha branch, change the metadata files and commit, the git actions workflow will validate.

Now the push has succeeded!

Rick@Dart MINGW64 ~/documents/github/plugins (Alpha)
$ git push origin Alpha
Enter passphrase for key '/c/Users/Rick/.ssh/id_ed25519':
Enter passphrase for key '/c/Users/Rick/.ssh/id_ed25519':
Enumerating objects: 18, done.
Counting objects: 100% (18/18), done.
Delta compression using up to 8 threads
Compressing objects: 100% (16/16), done.
Writing objects: 100% (16/16), 1.97 KiB | 1006.00 KiB/s, done.
Total 16 (delta 14), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (14/14), completed with 2 local objects.
To github.com:rgleason/plugins.git
   9cfacec..6fb681f  Alpha -> Alpha
rgleason commented 3 years ago

So rgleason/plugins/ Alpha branch is this now with the recent commit and push. https://github.com/rgleason/plugins/commit/6fb681f439e817a6dd543e49c415951feb6dc3bb

I have not made a PR to OpenCPN/plugins because it should not be necessary and will also remove the meta-url file too.

When I use OpenCPN now and update the Alpha Catalog, it looks like this

Screenshot (303)

and the trial version of weather_routing is not available yet.

bdbcat commented 3 years ago

So all is well? Or what?

rgleason commented 3 years ago

No. My weather_routing metadata does not show up as a choice in PIM plugin manager when under Alpha catalog and having updated that.

rgleason commented 3 years ago

This file was created automatically and it does not include the meta-url metadata. I thought it should be simply an ocpn-plugins.xml that includes just weather_routing metadata, but I don't know how the mechanizm works.

https://github.com/rgleason/plugins/blob/master/ocpn-plugins.xml

bdbcat commented 3 years ago

You mean Alpha branch, don't you?

Anyway, in the Alpha branch, on your repo, it looks like the commit "action" was skipped. In effect, this means that a fresh ocpn-plugins.xml was not created.

See here: https://github.com/rgleason/plugins/runs/3246360515

Check your settings->Actions in your github repo, Alpha branch.

bdbcat commented 3 years ago

Rick... I think I see why the action was skipped.

In your repo, Alpha branch:

  1. Edit file .github/workflows/CI_Merge.yml
  2. Remove the line: if: github.repository == 'OpenCPN/plugins'

Save and push the resulting file. You may then need to do a dummy push of a metadata file to trigger the action again.

rgleason commented 3 years ago

Dave, thanks. This works now. Later will try to document it.

Found Alpha WR 1.13.30.2 and installed it!

rgleason commented 3 years ago

The meta-url seems to be working well. I've pushed several versions up to my repos and it hass worked.