Obligations: Alignment with OSADL and OSLC

[mcjaeger]

(from discussion) existing data model should be aligned with the obligation data from these two sources.

(t.b.c)

[KoukiHama]

@mcjaeger

I would like to develop this feature.

So my question is: what does OSLC stand for?

Is OSDAL correct in the following link? https://www.osadl.org/Access-to-raw-data.oss-compliance-raw-data-access.0.html

Thanks Hama

[KoukiHama]

I found this pdf. OSLC stand for Open Source License Compliance. https://github.com/finos/OSLC-handbook/blob/master/output/pdf/OSLC-handbook.pdf

This is also good resource for importing license info. https://github.com/FHPythonUtils/LicenseMatrix

[tienlee]

@KoukiHama @mcjaeger

I would like to share the GUI and activity diagram.

Currently, I am implementing this feature following them. Next week, I will do the test and create a pull request.

Could you please help me to check and let me know your feedback?

Thanks and Best regards, TienL Import_OSADL_Licenses_Activity_Diagram.pdf

[mcjaeger]

@tienlee awesome! looks Ok for me. Please check that we have added to the sw360 (license) obligation data model a new field "external id" which could be used in this case to store a reference to the OSADL coordinates of that obligation. By this, your synchronisation steps could be supported.

[tienlee]

Thank you very much for your comment. Before, I plan to add a new field to the sw360 (license) obligation data model to store OSADL type. With your comment, I will use "external id" field. For example: "externalIds": { "OSADL-Obligation-URL": "https://www.osadl.org/fileadmin/checklists/unreflicenses/AFL-2.0.txt" },

[mcjaeger]

but pls consider the code status on latest master, maybe it will be complicated when doing this on tagged version 11 ...

[tienlee]

OK, I will consider it.

Currently, we are testing this feature. We got an issue when the internet speed is slow, the front end will be timeout, although the backend is in-progress to sync the data from osadl.org

Do you have any experience or suggestion for this case?

[KoukiHama]

This is the image of license obligations which sw360 import from OSDAL. @mcjaeger What do you think about this?

If you have any ideas on how to display OSDAL information more clearly, please let me know.

[mcjaeger]

Hello @KoukiHama that looks great. A quick idea could would be to make it two entries. (one for copyright notice, one of license text)

Also to import the obligation, maybe the Obligation title could be justth eline, so it can be reused between different licenses that share the same obligation. Otherwiese, if you have the ntp license, you will also have the same record, but checking on a project level will lead to checking a lot of same obligations.

So proposal could be:

1. Obligation: title "You must forward or provide copyright notices", Obligation text: "USER CASE Source code OR Binary. You must forward or provide copyright notices"
2. Obligation: title "You must forward or provide license text", Obligation text: "USE CASE Source code OR Binary. You must forward or provide license text"

So, this obligation record can be linked to many licenses...

[KoukiHama]

@mcjaeger Thank-you for good idea.

Our Image show MIT license case https://www.osadl.org/fileadmin/checklists/unreflicenses/MIT.txt

Other Licenses would be more complicated.

But, maybe, other license is also handled by your idea.

For example : LGPL2.1 https://www.osadl.org/fileadmin/checklists/unreflicenses/LGPL-2.1-only.txt

We should ignore some license obligation info in complicated licenses such as Comatibility in LGPL2.1-only.txt as first step. But these results will be helpful for users, even if sw360 does not show all OSDAL license info.

[KoukiHama]

Now I consider how to show LGPL 2.1 obligation from OSDAL.

if we use @mcjaeger 's proposal directly, we need to make obligation which have same title but different text obligation.

• Obligation: title "You must provide copyright notices", Obligation text: "USER CASE Source code delivery. You must provide copyright notices"

• Obligation: title "You must provide copyright notices", Obligation text: "USER CASE Binay delivery. IF NOT Linked work You must provide copyright notices"

I agree that one obligation record should be linked to many licenses.

So I think, as first step, we should make obligations about only YOU MUST or YOU MUST NOT in each license text by OSDAL texts.

(I think "USE CASE" or other "IF" requirement are also important, but we should consider them in next step. )

For example, MIT license has 2 obligations

• YOU MUST Forward OR Provide Copyright notices
• YOU MUST Forward OR Provide License text

Apache2 has these obligation and more.

• YOU MUST Forward License text
• YOU MUST Provide Modification notices
• YOU MUST Forward Copyright notices
• YOU MUST Forward Patent notices

LGPL 2.1 has these obligation and more.

• YOU MUST Provide Copyright notice
• YOU MUST Provide Warranty disclaimer
• YOU MUST NOT Modify License notices
• YOU MUST NOT Modify Warranty disclaimer
• YOU MUST Forward License text

NTL has these obligations

• YOU MUST Forward Copyright notices
• YOU MUST NOT Promote
• YOU MUST Forward Copyright notices In Documentation
• YOU MUST Forward License text In Documentation

PYTHON2.0 has these obligations

• YOU MUST Forward License text
• YOU MUST Forward Copyright notices
• YOU MUST Provide Modification report

By ignoring the complicated conditions, sw360 can link some common obligations for each license.

If we do this, the text and title might as well be the same.

Also, to avoid misleading users, information imported from OSDAL should be able to display its URL somewhere like SPDX links.

[quan-tv]

Dear @mcjaeger @KoukiHama,

This is my idea about the obligation:

• Firstly, I think that we should make clear the definition of obligation. I myself sometimes feel confused about it. I think that there are 2 different terms:

  • Obligation: one particular thing user need to follow when using the OSS. The construction of one obligation is: [Language element] + [Action] + [Object]. For example: [YOU MUST] + [Provide] + [Copyright notice] Ref: https://events19.linuxfoundation.org/wp-content/uploads/2018/07/OSLS-2019-Fulfilling-Open-Source-license-obligations-Can-checklists-help.pdf - Slide 7
  • License obligation: a set of obligations, also with the context provided (context is 'USE CASE', 'IF', ..) One obligation can be linked to multiple license obligation. But one license obligation can only be linked to one specific license.

• Import OSADL information:

  • The information from OSADL, follow the definition above, is license obligation.
  • SW360 needs to parse the OSADL information, break it into 'obligation' and 'context'. Attached PDF file is our first idea of how one license obligation is parsed and stored (please be noticed that this is just a sample). License Obligation.pdf
  • SW360 needs update current GUI for creating 'Obligation' and 'License Obligation'.

[KoukiHama]

@quan-tv Your idea is better than my tentative idea!

[quan-tv]

Hi all,

I and my colleague are working on parsing OSADL information from

https://www.osadl.org/Access-to-raw-data.oss-compliance-raw-data-access.0.html

We are facing a problem with parsing the obligation. In the last telco, I already explained about the structure of an obligation: [Language element] + [Action] + [Object]

For example:

Obligation: YOU MUST NOT Use "Apache" [ex. "Jakarta," "Apache," or "Apache Commons,"] in product name

• Language element: YOU MUST NOT
• Action: Use
• Object: "Apache" [ex. "Jakarta," "Apache," or "Apache Commons,"] in product name

Obligation: YOU MUST Credit Verbatim In Documentation "This product includes software developed by the Apache Software Foundation (http://www.apache.org/)."

• Language element: YOU MUST
• Action: Credit
• Object: Verbatim In Documentation "This product includes software developed by the Apache Software Foundation (http://www.apache.org/)."

While the 'Language element' can be pre-defined (at this moment, we noticed that there are only 2 values: YOU MUST and YOU MUST NOT), it is quite difficult to parse the Action and Object because they are free text. We came up with below solutions (none of them is perfect):

• Use 3rd library for parsing the text. For example: https://spacy.io/ . Library can be tested with this website: https://explosion.ai/demos/displacy? After a quick checking, I'm afraid that this library could not parse all the cases correctly.
• Pre-define a set of action words: Credit, Use, Forward, .. Of course we could not make sure that this set is still good in the future.
• Get the first word after 'Language element' as the 'Action'. This solution doesn't work for multi-word verbs.

So, if you have any other idea, could you please share with us?

[quan-tv]

Hi all,

I would like to continue with the design solution for OSADL information.

// I may need to break into multiple comments because wall of text.

Back to my previous comments about the 2 terms related to obligation:

• Obligation: one particular thing user need to follow when using the OSS.
  • License Obligation: a set of obligations, also with the context provided (context is 'USE CASE', 'IF', ..) One obligation can be linked to multiple license obligation. But one license obligation can only be linked to one specific license.

At this moment, SW360 already got a object named 'Obligation', but the meaning/purpose of this SW360 is the same meaning as 'License obligation' above. We should have 2 solutions here:

• Rename the 2 terms to make it fit with current SW360:
  • Obligation -> Obligation Element
  • License Obligation -> Obligation
• Rename current SW360 object:
  • Obligation -> License Obligation

Solution 1 doesn't require to update SW360 much, but the 2 new names doesn't reflex the meaning well. Solution 2 requires update the all the implemented source code related to Obligation.

I myself prefer the solution 2, because changing the name is not a difficult task, even though re-testing is still needed. But in case the re-testing effort is huge, we need to consider.

Could you please let me know your idea about this?

Thank you very much!

[quan-tv]

Next is parsing OSADL information. This is my proposal for solving this issue:

• Detect 'Language element' by checking if the key word 'YOU MUST', 'YOU MUST NOT' exist in the text or not.
• Use 'Stanza' (A.K.A 'StanfordNLP') for parsing the obligation text to get action and object. Github: https://github.com/stanfordnlp/stanza/ License: Apache-2.0 Even this library is quite reliable, we don't expect it could parse all the text perfectly. So, a new screen will be added for managing Obligation information, like below (draft version):

Obligation text from OSADL will be parsed and displayed this screen.

• Column 'Origin Statement': the original text from OSADL
• Column 'Language Element': basically got 1 of 2 values 'YOU MUST', 'YOU MUST NOT'
• Column 'Action': action of obligation
• Column 'Object': object of obligation
• Column 'Status': one of these values:
  • Not updated: parsed succefully by Stanza
  • Failed to define: Stanza could not parse the text
  • Updated: manually updated by user (also in this screen, when user clicks 'Edit', textboxes displayed for editting). (These values are still being considered)

[quan-tv]

For creating License Obligation, a new screen will replace the current screen of adding Obligation.

• Current screen of adding Obligation: user input Title, Text, Obligation Type, Obligation Level
• New screen: keep the Title, Obligation Type, Obligation Level, but the Text will not just a text box anymore. A new panel will be added, allow user to add text follow tree structure. For example, we got a part of 'GPL-2.0-or-later' license text like this:

USE CASE Source code delivery
    YOU MUST Provide Copyright notice
        ATTRIBUTE Highlighted
        ATTRIBUTE Appropriately
    IF Software modification
        YOU MUST Grant License
USE CASE Binary delivery
COMPATIBILITY BSD-2-Clause
COPYLEFT CLAUSE Yes

The corresponding screen is like:

• Root node: there is only one Root note (textbox), and the name of the license should be input here.
• Node: a node will reflex a statement in the license obligation. 'Type': mandatory. Depend on which value is selected or inputted, we will display one or multiple textboxes, combo boxes, allow user to input detail information. For example:
  • If type is 'OBLIGATION', user will need to input Language element, Action, Object (or select from Obligation table - my previous comment)
  • If type is 'ATTRIBUTE', user will need to input 'Text' only.
  • If type is 'OPERATOR', user will need to input 'Operator' and Text
  • Beside pre-defined values, user could freely input new values of type, for example: ANY THING. Cause this value will be stored in DB, it will displayed in the combo boxes the next time user uses this function - therefore not only convenience for user, but also allow to support new values in the future.
• Node actions: depend on which node is focused, the corresponding actions button will be displayed
  • Root node: user can only add child node
  • Other nodes: user can add sibling nodes, child nodes or delete (when a node is deleted, all of its child nodes will also be deleted)

[quan-tv]

By the way, I have a question. I notice that there is no 'Edit' function of Obligation in current SW360. I wonder if this is the intention when design the system or just a missing feature?

If case if it is a missing feature, I and my colleagues would like to implement it.

[quan-tv]

Hi all,

About this comment of mine about the terms related to Obligation:

https://github.com/eclipse/sw360/issues/513#issuecomment-877928926

We should have 2 solutions here:

• Rename the 2 terms to make it fit with current SW360:
  • Obligation -> Obligation Element
  • License Obligation -> Obligation
• Rename current SW360 object:
  • Obligation -> License Obligation

After deeper investigation about current SW360 system, I found out that the term 'License Obligation' was used for 'Obligation Level', so I'm afraid that the solution 2 is not feasible.

So, should we follow the solution 1?

[KoukiHama]

I see. Sw360 have already used "License Obligation". So, I think only solution 1 is good.

[mcjaeger]

By the way, I have a question. I notice that there is no 'Edit' function of Obligation in current SW360. I wonder if this is the intention when design the system or just a missing feature?

If case if it is a missing feature, I and my colleagues would like to implement

We thought that once obliugations are there, they should not be changed after wards. Maybe some text type correction. Bu tthe problem is when you edit an existing oibligation how to follow later if an obligation was covered by the product?

However, I think the obligation data is copied to the project for preserving what is considered b the projects (I do not remember exactly anymore). So the obligation text in the main database could be edited in fact.

[mcjaeger]

Hi all,

I would like to continue with the design solution for OSADL information.

// I may need to break into multiple comments because wall of text.

Back to my previous comments about the 2 terms related to obligation:

• Obligation: one particular thing user need to follow when using the OSS.
• License Obligation: a set of obligations, also with the context provided (context is 'USE CASE', 'IF', ..) One obligation can be linked to multiple license obligation. But one license obligation can only be linked to one specific license.

At this moment, SW360 already got a object named 'Obligation', but the meaning/purpose of this SW360 is the same meaning as 'License obligation' above. We should have 2 solutions here:

• Rename the 2 terms to make it fit with current SW360:

  • Obligation -> Obligation Element
  • License Obligation -> Obligation

• Rename current SW360 object:

  • Obligation -> License Obligation

Solution 1 doesn't require to update SW360 much, but the 2 new names doesn't reflex the meaning well. Solution 2 requires update the all the implemented source code related to Obligation.

I myself prefer the solution 2, because changing the name is not a difficult task, even though re-testing is still needed. But in case the re-testing effort is huge, we need to consider.

Could you please let me know your idea about this?

Thank you very much!

I think the idea was not to have a container kind of type, but to have a type hierarchy. There shall be the super type Obligationwhere the sub types inheriting from that are

• Licensse Obligation inheriting from licenses,
• Component Obligation being special about a component (for example an ECC issue),
• Project obligation, for example all projects from a special business area need a special OSS handling
• Organisation Obligation, for example valid for all projects or products in the organisation, provide an OSS office contact address.

For the OSADL obligations we need probably license obligations, because all of them refer to a license.

[KoukiHama]

https://www.osadl.org/fileadmin/checklists/matrix.html

This is good matrix for updating sw360's OSDAL function. For example, Alert function if license compatibility have problrm.

[quan-tv]

Dear all, We are working to implement new functions of Obligation management:

• Edit one obligation
• Duplicate one obligation
• View obligation changelog

Here are some screen shots of these functions:

• Obligation list screen:
• Edit one obligation: similar to the screen 'Add obligation'
• Duplicate one obligation: re-use the screen 'Add obligation'
• View obligation changelog: similar to changelog screen of Project, Component, Release

Actually, the implementation was completed, we are in testing phase.

I hope that these functions will be ready soon.

[KoukiHama]

Lets continue to discussion about Import OSLC

[arunazhakesan]

PR is merged.