search

spdx / LicenseListPublisher

Tool that generates license data found in the license-list-data repository from the license-list-XML source
Apache License 2.0
11 stars 18 forks source link

Multi-threading #71

Closed tjasmith closed 4 years ago

tjasmith commented 4 years ago
tjasmith commented 4 years ago

@goneall Whenever I try to get the crossref from the license for the mustachmap, I have this error during the execution:

Template error for license HTML file: Failed to execute method: public boolean org.spdx.htmltemplates.LicenseHTMLFile$FormattedUrl.getIsLive()

Probably comes from the SpdxWebsiteFormatWriter, line 215 or from the LicenseRdfaFormatWriter, line 167

Any idea how to solve this?

While working to find a solution for this, I have commented code that caused this.

tjasmith commented 4 years ago

@goneall With this update, even in the event of a timeout(due to a call to check the live status of the url), the crossrefs details are set, with the isLive parameter set to False.

goneall commented 4 years ago

With this update, even in the event of a timeout(due to a call to check the live status of the url), the crossrefs details are set, with the isLive parameter set to False.

@tjasmith I had a few performance improvement suggestions. I just ran this against the entire license list and I noticed it did serialize on invalid URL's impacting the performance, so I'm pretty sure it would benefit from some minor redesign. That being said, it completed in 20 minutes which I think is reasonable for now. I also found an issue completely unrelated to your code: https://github.com/spdx/LicenseListPublisher/issues/72 which could improve performance quite a bit.

We can create a separate issue to implement the performance improvements later if you would like me to merge this PR - just let me know.

tjasmith commented 4 years ago

@goneall I think it's best this PR is merged and another issue is created for further improvements.

goneall commented 4 years ago

Added an issue to track the performance improvement: https://github.com/spdx/LicenseListPublisher/issues/73