Changes in website format prevents pulling changelog when generating Debian package

[aureq]

Hi @mattab

It seems like the website format (HTML) has change and prevents debian/scripts/history.sh from pulling the history out of the official site.

We have 3 options:

1. The HTML code for the site returns to what it used to be (one <li>...</li> per line) to allow the script to create the changelog file
2. Matomo writes a clean issue changelog in https://github.com/matomo-org/matomo/releases as part of publishing a new release and the script will pull the changelog from there instead of matomo.org
3. Matomo provides a json or text file the script is able to easily parse to generate the changelog

Cheers

[mattab]

Hi @aureq It looks like there is still one <li> per line on here: view-source:https://matomo.org/changelog/matomo-3-8-0/ or maybe we changed something. what is the issue exactly?

[aureq]

The issue is that HTML elements are broken over multiple lines.

</li><li><a
href="https://github.com/matomo-org/matomo/issues/13419">#13419</a> Same icon is used for &#8220;segmented visitor log&#8221; and &#8220;show visitor profile&#8221; [by <a
href="https://github.com/tsteur">@tsteur</a>]</li><li><a
href="https://github.com/matomo-org/matomo/issues/13005">#13005</a> Make log and report data screen less technical [by <a
href="https://github.com/tsteur">@tsteur</a>]</li><li><a
href="https://github.com/matomo-org/matomo/issues/13328">#13328</a> When you are logged out, the URL gets lost when you log in [by <a
href="https://github.com/tsteur">@tsteur</a>]</li><li><a
href="https://github.com/matomo-org/matomo/issues/13456">#13456</a> Locations users country map should use Unique visitors by default [by <a
href="https://github.com/sgiehl">@sgiehl</a>]</li><li><a
href="https://github.com/matomo-org/matomo/issues/13597">#13597</a> When token_auth is re-generated, remove wording that user will be logged out  [by <a
href="https://github.com/tsteur">@tsteur</a>]</li><li><a
href="https://github.com/matomo-org/matomo/issues/1781">#1781</a> New diagnostic checking that max_allowed_packet is at least 64M or issue warning, and recommend 128M [by <a
href="https://github.com/tsteur">@tsteur</a>]</li><li><a

As you can see above, <li> doesn't start at the beginning of a line and <a href= spans 2 lines.

Now, I could adapt the script to deal with the changes but I feel like it's an extra effort and it will require changes in the future if the HTML changes in the future (it did a few times in the past).

I think that having a json, text, csv file part of Matomo source code could be a great (and long term) solution. This would allow Matomo to display all the latest changes as part of a user installed Matomo instance, make it easy for the package generator to embed the changes in the Debian package, and quite easy to embed the data in the Matomo release page too.

Or, if we don't care to embed the changelog inside the page, we could just point to a URL for each release. But that's not in the spirit of a changelog.

What do you think?

PS: In a shorter form, the more predictable our processes are the better so I can fully automate the package process.

[Findus23]

@aureq Maybe you could take the data directly from the changelog generator. You could also take a look at my fork that creates markdown and HTML based on that markdown (https://github.com/Findus23/github-changelog-generator), but I don't promise there aren't any bugs.

Maybe when I have too much time in my holiday (in a few weeks), I could rewrite it as a python script which shouldn't be that hard (and have fewer issues with synchronous requests).

[aureq]

@Findus23 Have you thought of using github API to pull down a list of issues via https://developer.github.com/v3/issues/ ?

As an idea, it might be possible to filter down a list of resolved/closed issues that are between 2 GIT tags (which would be a start and a stop date). From there, it would then be possible to pull the list of issues that are marked as closed, generate any json/changelog or anything?

What do you think ?

[Findus23]

@aureq That's exactly what the github-changelog-generator does (apart being able to select tags instead of dates, which I think is a great idea)

[Findus23]

@aureq I had some time and this is the result: https://github.com/Findus23/new-github-changelog-generator

It's not perfect and needs some more testing, but it can even directly generate debian changelogs.

  [ matomo-org/matomo ]
  * Encoding issue with failed update [by @diosmosis] (Closes: #13626)
  * Improve the style of the visitor log [by @sgiehl] (Closes: #13332)
  * Make Matomo PHP 7.3 compatible(Closes: #13418)
  * Let plugins change the report type in sendReport event [by @tsteur,
    @diosmosis] (Closes: #14098)
  * Sales reports: Segmented visitor log and row evolution should segment
    users by ecommerce orders [by @diosmosis] (Closes: #10989)
  * Visitor Map tooltip shows %s [by @tsteur] (Closes: #14054)
  * Document the new Ecommerce JavaScript Tracker function
    'getEcommerceItems' [by @diosmosis] (Closes: #14035)
  * Show actions in visitorlog as a (unnumerated) timeline [by @diosmosis,
    @sgiehl] (Closes: #13916)
  * Number of database upgrades off [by @diosmosis] (Closes: #13992)
  * Add possibility to change mail transport through DI [by @tsteur,
    @diosmosis] (Closes: #14041)
  * Use absolute path for alternative tracker file [by @tsteur, @diosmosis]
    (Closes: #14059)
  * Typo in FR language [by @acs-ferreira] (Closes: #14061)
  * Optimize all archive tables only monthly [by @tsteur, @diosmosis]
    (Closes: #14068)
  * Remember me - not working(Closes: #14075)

  [ matomo-org/tag-manager ]
  * Custom JS Function isn't working when using multiple custom
    functions(Closes: #148)
  * Fix greater than comparison is not working [by @tsteur] (Closes: #151)

[aureq]

Closing issue.

[mattab]

Hi @aureq fyi I am not using the new github changelog generator but the old one. let me know if there is some issue with the old one (in JS) as i'd like to fix it in there rather than use the new one...