search

spdx / Spdx-Java-Library

Java library which implements the Java object model for SPDX and provides useful helper functions
Apache License 2.0
37 stars 36 forks source link

[Question] About the parameter of writeToFile method and readFile method #66

Closed shi9qiu closed 2 years ago

shi9qiu commented 3 years ago

Hello,

My team is working on enhancing the functionalities of SW360 (https://github.com/eclipse/sw360) - a software component catalog application. We plan to replace the old library used in SW360 with this library. But we are facing some problems so I come here to see if I can get some help.

We are using writeToFile method (https://github.com/spdx/Spdx-Java-Library/blob/master/src/main/java/org/spdx/library/Read.java#L124) and readFile method (https://github.com/spdx/Spdx-Java-Library/blob/master/src/main/java/org/spdx/library/Write.java#L97) to implement the Import/Export functions of SW360. However, these methods need one parameter modelStore, but as we understand, this interface was not implemented yet. Ref: https://github.com/spdx/Spdx-Java-Library/blob/42fcd5bc77ee49afc1bfb2de992679162bbcb3cd/src/main/java/org/spdx/storage/ISerializableModelStore.java#L31

Could you tell me if our understanding is correct or not?

Another question is, we found this message in the README.md: The default storage interface is an in-memory Map which should be sufficient for light weight usage of the library.

Does it mean we cannot use this library to read or write the large size file?

Thank you in advance!

goneall commented 3 years ago

@shi9qiu I'm quite happy to help support the use of this library in SW360.

However, these methods need one parameter modelStore, but as we understand, this interface was not implemented yet.

There are 3 implementations of this interface, but they are in different Github repos - you can add them as dependencies to your project. The 3 implementations represent 3 different serialization formats for SPDX:

Does it mean we cannot use this library to read or write the large size file?

I've actually used this store for some very large files without issue - it does store everything in memory, however, so I'm sure there will be a limit. I have a prototype RDF graph database implementation for very large RDF graphs. I've thought about writing an SQL based store, but haven't found the need yet.

goneall commented 3 years ago

@shi9qiu If you would like an example using this library with the above mentioned store, check out the SPDX Converter utility

shi9qiu commented 3 years ago

Thank you so much for your speedy reply! We will try it.

nam-np commented 3 years ago

Hello, @goneall I used this function createSpdxDocument() for make SpdxDocument object from RDF file input, but it is seem not work with large file. In library Spdx-Java-Library has same function can work with large file?

nam-np commented 3 years ago

And a sample RDF large file https://github.com/eclipse/sw360/files/5984378/SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf.zip

nam-np commented 3 years ago

I re-checked the library spdx-tools with function Verify and it used same function createSpdxDocument for making SpdxDocument. I tested with RDF large file https://github.com/eclipse/sw360/files/5984378/SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf.zip (~80MB) and it can work fine.

namnp@namnp:~/tools$ java -jar target/spdx-tools-2.2.6-SNAPSHOT-jar-with-dependencies.jar Verify SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf/SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf 
--------Start verify: [SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf/SPDX2_linux-4.18.tar.xz_1535128294-spdx.rdf]
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'log4j2.debug' to show Log4j2 internal initialization logging.
15:55:40.523 [main] ERROR org.spdx.rdfparser.license.ListedLicenses - I/O error opening Json TOC URL, using local TOC file
spdx.org
spdx.org
spdx.org
...
--------getSpecVersion: SPDX-2.1
--------getSpecName:   /srv/fossology/repository/report
This SPDX Document is valid.

So, I will re-check when using function with SW360.

goneall commented 2 years ago

@nam-np Were you able to re-check with SW360? I'm working on a minor release for this library and would like to work in fixes for this if needed.

KoukiHama commented 2 years ago

@goneall @nam-np Some members in SW360 community tried new library,

We got error again. https://github.com/eclipse/sw360/issues/1171#issuecomment-990610778 https://github.com/eclipse/sw360/issues/1171#issuecomment-990612685

But we found a root problem. https://github.com/eclipse/sw360/issues/1171#issuecomment-990614065

goneall commented 2 years ago

@KoukiHama Thanks for the update - I left a comment in the SW360 issue.

Let me know if you'd like any help migrating from the old SPDX tools library to this one (I've done it a few times).

Since the issue is an incompatibility between the old library and this library, I'm going to close out this issue since I can't think of any way to resolve it without re-introducing a security vulnerability. If you disagree or know of a way to resolve it with a change to this issue, please add a comment.