search

eclipse-embed-cdt / eclipse-plugins

The Eclipse Embedded CDT plug-ins for Arm & RISC-V C/C++ developers (formerly known as the GNU MCU Eclipse plug-ins). Includes the archive of previous plug-ins versions, as Releases.
http://eclipse-embed-cdt.github.io/
Eclipse Public License 2.0
558 stars 130 forks source link

Add p2.mirrorsURL and p2.statsURI to the repository #395

Closed ilg-ul closed 4 years ago

ilg-ul commented 4 years ago

The Eclipse Foundation download infrastructure requires two properties in the repositories.

In CDT they are added with an ant task and a xsl file.

The documentation page is

Find a similar solution for Embed CDT.

ilg-ul commented 4 years ago

@ruspl-afed, now that we have an updated Tycho, do you know how to add these two properties to the p2 repository?

ilg-ul commented 4 years ago

I tried to add the following to ilg.gnumcueclipse.repository/pom.xml, but I could not find the result in the output, there must be some other trick that I missed:

    <build>
        <plugins>
        <plugin>
            <groupId>org.eclipse.tycho</groupId>
            <artifactId>tycho-p2-repository-plugin</artifactId>
            <version>${tycho.version}</version>
            <configuration>
            <extraArtifactRepositoryProperties>
                <p2.statsURI>http://download.eclipse.org/stats</p2.statsURI>
                <p2.mirrorsURL>http://www.eclipse.org/downloads/download.php?file=/embed-cdt/updates/neon&amp;format=xml</p2.mirrorsURL>
                <foo>bar</foo>
            </extraArtifactRepositoryProperties>
            </configuration>
        </plugin>
        </plugins>
    </build>
ilg-ul commented 4 years ago

Here is another documentation page:

ruspl-afed commented 4 years ago

This is one can be a big task from what I can read around. I'm not happy with solutions that I can find at the moment - too much dummy work. My plan was to copy the solution from CDT, but after reading its scripts I'm no longer sure that CDT sets the p2.mirrorsURL properly. @jonahgraham do you have any updates for Bug 536339 ?

ilg-ul commented 4 years ago

I'm no longer sure that CDT sets the p2.mirrorsURL properly

that's not very encouraging. :-(

is there any plug-in known to have the mirrors set properly?

ilg-ul commented 4 years ago

Hi Alexander, any progress on this?

I recently had difficulties to install CDT due to a routing problem in Germany, the p2 install always used that mirror and the only way out was to bring the repository archive locally:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=565861

I'm not sure if this could be avoided by a better CDT mirror configuration, but if there is anything we can do to avoid such cases, I think we should.

The plan is to release 5.1.2 with 2020-09, possibly with Aarch64 support, if the packing issues are solved by then.

jonahgraham commented 4 years ago

CDT does not set up mirrors in p2 - now that download.eclipse.org mirrors with redirects it is less relevant and I do wonder if it is still a real requirement.

ilg-ul commented 4 years ago

I do wonder if it is still a real requirement.

if mirrors in p2 are no longer required, how about analytics? is p2.statsURI useful?

ilg-ul commented 4 years ago

CDT does not set up mirrors in p2

Isn't this still applied?

https://github.com/eclipse-cdt/cdt/blob/00a52086c9e3c7457be830ee38626c2af0225c23/releng/org.eclipse.cdt.repo/antArtifactsUpdater.xml#L38-L40

This seems in accordance with the wiki, mentioned before.

ruspl-afed commented 4 years ago

@ilg-ul , to be honest I do not understand quite good the internals behind the "download" facade, so I cannot state that we need to set a particular property to a particular value and everything will start to function as expected. I hope @jonahgraham can help us to formulate what is really needed to be done as he is much closer to this with EPP project, or probably redirect us to the appropriate person/resource to learn the state of the art.

ilg-ul commented 4 years ago

Do you know if the code used in CDT (ant + xslt) does the trick of setting those two properties in the final p2, as documented in the wiki?

ruspl-afed commented 4 years ago

1) I can see an attempt to change only one property mirrorsURL 2) I do not see where this ant script is used by CDT builds this is why I asked @jonahgraham to comment on this.

jonahgraham commented 4 years ago

Do you know if the code used in CDT (ant + xslt) does the trick of setting those two properties in the final p2, as documented in the wiki?

Those files are left there - but no one in CDT uses them AFAIK. They were supposed to make those settings, but they haven't been used by CDT from long before I took over releng.

See https://bugs.eclipse.org/bugs/show_bug.cgi?id=561651 for details on how EPP does it.

ilg-ul commented 4 years ago

If Jonah is happy with this solution for CDT, it should work for Embed CDT too.

Is there a page to see the download figures for CDT p2?

I guess the trick is done by <generateDownloadStatsProperty>true</generateDownloadStatsProperty>.

The URI statistics seems to be common for all EPP packages: <p2.statsURI>http://download.eclipse.org/stats/technology/epp/packages/${eclipse.simultaneous.release.id}/</p2.statsURI>, we need to update this to a project specific one.

ilg-ul commented 4 years ago

I added the two Tycho properties and they are passed to the generated repo, so I guess this should fix this issue.

[edited after fixing the Jenkins issue]

jonahgraham commented 4 years ago

Btw - edits to Github don't seem to generate email notifications, unfortunately I started investigating the now edited out comment.

For the record, and anyone else following this issue, @ilg-ul solved tee problem and https://ci.eclipse.org/embed-cdt/job/build-plug-ins/job/develop/27/changes appears to have the changes

ilg-ul commented 4 years ago

I apologise for the false alarm, this time it was not Jenkins fault, it was my mistake. :-(

I'm still working on documenting the build and publish procedure, and the first draft was incomplete.

To avoid such cases, I added a README-MAINTAINER.md, with a detailed checklist. It is still work in progress, but I hope it is a good start.

The build passed, I'll test the result tomorrow, and if everything is fine, publish 5.1.2 soon.

ilg-ul commented 4 years ago

For reference, here are the doc pages that I used:

The 'download_stats' link is mentioned in Eclipse Download Stats page.

ilg-ul commented 4 years ago

I just found out how to check the p2 statistics. For example for projects that start with 'c', use

https://dev.eclipse.org/committers/committertools/stats.php?filename=/stats/c

As you can see, CDT does not leave p2 statistics. :-(

Unfortunately neither my plug-ins, although I already set the p2.statURI property. :-(

https://dev.eclipse.org/committers/committertools/stats.php?filename=/stats/embed-cdt

There must be some other trick that I missed.

I was told that to ask on cross-project-issues-dev. Any other suggestions on how to proceed?

ilg-ul commented 4 years ago

After studying the docs more thoroughly, I discovered that I also have to set <generateDownloadStatsProperty>true</generateDownloadStatsProperty> in tycho.

This automatically generates the <property name='download.stats' value='XXXXX'/> for all features.

ilg-ul commented 4 years ago

The p2 stats are finally visible in the Eclipse Download Stats page:

https://dev.eclipse.org/committers/committertools/stats.php?filename=/stats/embed-cdt

However, I'm not particularly happy with the large number of records generated for a single install. All 42 features and plug-ins left a line in the server.

In my opinion, since during install the user can only select which features to install, only the feature records are necessary, and the plug-in records are mostly redundant.

Also the feature names (ilg.gnumcueclipse.debug.gdbjtag.qemu.feature.feature.jar/3.4.1.202008191141), with the extra .feature.jar, seem not very inspired.

Any comments?

jonahgraham commented 4 years ago

The p2 stats are finally visible in the Eclipse Download Stats page:

https://dev.eclipse.org/committers/committertools/stats.php?filename=/stats/embed-cdt

Great! I am going to try to adapt what you figured out back to CDT as it looks useful and simple enough.

However, I'm not particularly happy with the large number of records generated for a single install. All 42 features and plug-ins left a line in the server.

A slightly more refined query should help I suppose. Others may have more experience. Mickael Istria has had numerous discussions about stats on cross-project-issues-dev in the past.

In my opinion, since during install the user can only select which features to install, only the feature records are necessary, and the plug-in records are mostly redundant.

That is true for direct installs, but for end users who install a non embed-cdt feature that only references some bundles from embed-cdt those stats would not exist at all.

Also the feature names (ilg.gnumcueclipse.debug.gdbjtag.qemu.feature.feature.jar/3.4.1.202008191141), with the extra .feature.jar, seem not very inspired.

I think that is the standard across eclipse, i.e. that is what appears in the .target files IIRC

Any comments?

Summary -> :-) Wonderful getting the stats enabled. I hope it provides you useful info in the future!

One question I have - and perhaps you have discovered the answer now. If you contribute embed-cdt to simrel, do you have two sets of stats to consider? The jars in simrel and the jars in your project repo? Or does the p2 magic combine all of that?

ilg-ul commented 4 years ago

So, @jonahgraham, your suggestion is to keep the current configuration? (I like when I don't have to do any more work...)

If you contribute embed-cdt to simrel, do you have two sets of stats to consider? The jars in simrel and the jars in your project repo? Or does the p2 magic combine all of that?

Good question! The current p2 repo is generated by a clone of org.eclipse.epp.packages where I added my code, and my code is a clone of CDT with several additions.

It is a maintenance nightmare and I would be grateful for any suggestions on how to simplify/automate things.

To answer your question, I guess that the jars will remain the same, so the stats will remain the same, they are per project, but I'm not sure I understand all details.

ilg-ul commented 4 years ago

Seems ok, p2 stats available at:

https://dev.eclipse.org/committers/committertools/stats.php?filename=/stats/embed-cdt

ilg-ul commented 4 years ago

I did a local maven run for my local EPP, right after publishing a new version of the plug-ins on /updates/neon-test, so without giving the mirrors time to propagate the updates.

The result was a series of warnings:

[WARNING] Some attempts to read artifact osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 failed:
[WARNING]    An error occurred while transferring artifact canonical: osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 from repository https://download.eclipse.org/embed-cdt/updates/neon-test:
[WARNING]       Retry another mirror:
[WARNING]          Artifact not found: http://ftp.fau.de/eclipse/embed-cdt/updates/neon/plugins/ilg.gnumcueclipse.debug.core_1.2.2.202008280727.jar.
[WARNING]    An error occurred while transferring artifact canonical: osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 from repository https://download.eclipse.org/embed-cdt/updates/neon-test:
[WARNING]       Retry another mirror:
[WARNING]          Artifact not found: http://ftp.fau.de/eclipse/embed-cdt/updates/neon/plugins/ilg.gnumcueclipse.debug.core_1.2.2.202008280727.jar.
[WARNING]    An error occurred while transferring artifact canonical: osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 from repository https://download.eclipse.org/embed-cdt/updates/neon-test:
[WARNING]       Retry another mirror:
[WARNING]          Artifact not found: http://ftp.fau.de/eclipse/embed-cdt/updates/neon/plugins/ilg.gnumcueclipse.debug.core_1.2.2.202008280727.jar.
[WARNING]    An error occurred while transferring artifact canonical: osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 from repository https://download.eclipse.org/embed-cdt/updates/neon-test:
[WARNING]       Retry another mirror:
[WARNING]          Artifact not found: http://ftp-stud.fht-esslingen.de/pub/Mirrors/eclipse/embed-cdt/updates/neon/plugins/ilg.gnumcueclipse.debug.core_1.2.2.202008280727.jar.
[WARNING]    An error occurred while transferring artifact canonical: osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 from repository https://download.eclipse.org/embed-cdt/updates/neon-test:
[WARNING]       Retry another mirror:
[WARNING]          Artifact not found: http://ftp.fau.de/eclipse/embed-cdt/updates/neon/plugins/ilg.gnumcueclipse.debug.core_1.2.2.202008280727.jar.
[WARNING]    An error occurred while transferring artifact canonical: osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 from repository https://download.eclipse.org/embed-cdt/updates/neon-test:
[WARNING]       Retry another mirror:
[WARNING]          Artifact not found: http://ftp.snt.utwente.nl/pub/software/eclipse//embed-cdt/updates/neon/plugins/ilg.gnumcueclipse.debug.core_1.2.2.202008280727.jar.
[WARNING]    An error occurred while transferring artifact canonical: osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 from repository https://download.eclipse.org/embed-cdt/updates/neon-test:
[WARNING]       Retry another mirror:
[WARNING]          Artifact not found: http://mirror.dkm.cz/eclipse/embed-cdt/updates/neon/plugins/ilg.gnumcueclipse.debug.core_1.2.2.202008280727.jar.
[WARNING]    An error occurred while transferring artifact canonical: osgi.bundle,ilg.gnumcueclipse.debug.core,1.2.2.202008280727 from repository https://download.eclipse.org/embed-cdt/updates/neon-test:
[WARNING]       Retry another mirror:
[WARNING]          Artifact not found: http://ftp-stud.fht-esslingen.de/pub/Mirrors/eclipse/embed-cdt/updates/neon/plugins/ilg.gnumcueclipse.debug.core_1.2.2.202008280727.jar.
[INFO] Fetching ilg.gnumcueclipse.debug.gdbjtag_4.2.1.202008280727.jar from https://download.eclipse.org/embed-cdt/updates/neon-test/plugins/ (371.75kB)

In other words, the logic to attempt several retries is functional, there were 8 attempts before using the direct URL.

Since I don't remember seeing this before, and I don't know exactly what p2.mirrorsURL does, I infer that this might have helped. :-)