search

ros-infrastructure / superflore

An extended platform release manager for Robot Operating System
Apache License 2.0
51 stars 33 forks source link

License parsing doesn't parse e.g. LGPLv3 correctly #271

Closed shr-project closed 3 years ago

shr-project commented 4 years ago

When regenerating recipes for newer eloquent release from 2020-01-24, I've noticed that generated mavlink recipe has license LGPL-2 while the original package.xml has LGPLv3 (https://github.com/mavlink/mavlink-gbp-release/blob/release/eloquent/mavlink/2019.12.30-1/package.xml#L20)

Which for OE triggers this warning:

WARNING: mavlink-2019.12.30-1-r0 do_populate_lic: mavlink: No generic license file exists for: LGPL-2 in any provider

because LGPL-2 isn't valid SPDX identifier (https://spdx.org/licenses/) nor OE licenses.conf has mapping from LGPL-2 to the valid LGPL-2.0 (http://git.openembedded.org/openembedded-core/tree/meta/conf/licenses.conf#n28)

Mapping LGPLv3 to LGPL-2 is quite wrong and dangerous. I've tried to update the regexps here: https://github.com/ros-infrastructure/superflore/blob/master/superflore/utils.py#L149 but that's also a bit error prone with the free form of license identifiers people are using in package.xml.

What do you think about using exact mapping for known identifier variants, like licenses.conf in OE does http://git.openembedded.org/openembedded-core/tree/meta/conf/licenses.conf and then mapping them only to valid SPDX identifiers or issuing the warning and passing the value when it's not one of the explicitly supported variants?

That would also cover the issue that LGPL (without any version) is currently being mapped to this LGPL-2 which is very misleading when the actual license in the component is LGPL-3.0.

I have already couple cases in meta-ros (mostly from melodic release) where this happens:

meta-ros1-melodic/recipes-bbappends/navigation/amcl_1.16.3-1.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/navigation/amcl_1.16.3-1.bbappend-LICENSE = "LGPL-2.1+"
meta-ros1-melodic/recipes-bbappends/navigation/amcl_1.16.3-1.bbappend-
--
meta-ros1-melodic/recipes-bbappends/open-karto/open-karto_1.2.0.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/open-karto/open-karto_1.2.0.bbappend-LICENSE = "LGPL-3.0"
--
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_1.4.0.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_1.4.0.bbappend-# https://github.com/orocos/orocos-kdl-release/issues/4
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_1.4.0.bbappend-LICENSE = "LGPL-2.1"
--
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/orocos-kinematics-dynamics_1.4.0.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/orocos-kinematics-dynamics_1.4.0.bbappend-# https://github.com/orocos/orocos-kdl-release/issues/4
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/orocos-kinematics-dynamics_1.4.0.bbappend-LICENSE = "LGPL-2.1"
--
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/python-orocos-kdl_1.4.0.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/python-orocos-kdl_1.4.0.bbappend-# https://github.com/orocos/orocos-kdl-release/issues/4
meta-ros1-melodic/recipes-bbappends/orocos-kinematics-dynamics/python-orocos-kdl_1.4.0.bbappend-LICENSE = "LGPL-2.1"
--
meta-ros1-melodic/recipes-bbappends/rqt-multiplot-plugin/rqt-multiplot_0.0.10.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/rqt-multiplot-plugin/rqt-multiplot_0.0.10.bbappend-# https://github.com/ANYbotics/rqt_multiplot_plugin/issues/35
meta-ros1-melodic/recipes-bbappends/rqt-multiplot-plugin/rqt-multiplot_0.0.10.bbappend-LICENSE = "LGPL-3.0"
--
meta-ros1-melodic/recipes-bbappends/slam-karto/slam-karto_0.8.1.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/slam-karto/slam-karto_0.8.1.bbappend-# https://github.com/ros-perception/slam_karto/issues/28
meta-ros1-melodic/recipes-bbappends/slam-karto/slam-karto_0.8.1.bbappend-LICENSE = "LGPL-3.0+ & CC-BY-2.0"
--
meta-ros1-melodic/recipes-bbappends/variant/variant-msgs_0.1.5.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/variant/variant-msgs_0.1.5.bbappend-# https://github.com/ANYbotics/variant-release/issues/1
meta-ros1-melodic/recipes-bbappends/variant/variant-msgs_0.1.5.bbappend-LICENSE = "LGPL-3.0+"
--
meta-ros1-melodic/recipes-bbappends/variant/variant-topic-test_0.1.5.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/variant/variant-topic-test_0.1.5.bbappend-# https://github.com/ANYbotics/variant-release/issues/1
meta-ros1-melodic/recipes-bbappends/variant/variant-topic-test_0.1.5.bbappend-LICENSE = "LGPL-3.0+"
--
meta-ros1-melodic/recipes-bbappends/variant/variant-topic-tools_0.1.5.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/variant/variant-topic-tools_0.1.5.bbappend-# https://github.com/ANYbotics/variant-release/issues/1
meta-ros1-melodic/recipes-bbappends/variant/variant-topic-tools_0.1.5.bbappend-LICENSE = "LGPL-3.0+"
--
meta-ros1-melodic/recipes-bbappends/variant/variant_0.1.5.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros1-melodic/recipes-bbappends/variant/variant_0.1.5.bbappend-# https://github.com/ANYbotics/variant-release/issues/1
meta-ros1-melodic/recipes-bbappends/variant/variant_0.1.5.bbappend-LICENSE = "LGPL-3.0+"
--
meta-ros2-dashing/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_3.2.1-1.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros2-dashing/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_3.2.1-1.bbappend-# https://github.com/ros2-gbp/orocos_kinematics_dynamics-release/issues/2
meta-ros2-dashing/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_3.2.1-1.bbappend-LICENSE = "LGPL-2.1"
--
meta-ros2-eloquent/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_3.2.1-1.bbappend:# Missing license version in package.xml maps it to invalid LGPL-2
meta-ros2-eloquent/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_3.2.1-1.bbappend-# https://github.com/ros2-gbp/orocos_kinematics_dynamics-release/issues/2
meta-ros2-eloquent/recipes-bbappends/orocos-kinematics-dynamics/orocos-kdl_3.2.1-1.bbappend-LICENSE = "LGPL-2.1"
shr-project commented 4 years ago

Here are some statistics from current rosdistro (across package.xml files in all ROS distributions):

      1 WARNING: The license value "Apache License Version 2.0" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "Apachi 2" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "BSD 3-clause Clear License" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "BSD 3-clause. See license attached" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "CreativeCommons-Attribution-NonCommercial-NoDerivatives-4.0" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "CreativeCommons-Attribution-NonCommercial-ShareAlike-4.0-International" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "FZI all rights reserved" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "GNU GPL v3.0" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "GPL + runtime exception" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "GPL v2 with linking exception" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "GPL v3" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "GPL-2.0" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "LGPL / BSD" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "LGPL and Apache2" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "LGPL v2" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "LGPL v2.1 or later" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "LGPLv2.1, modified BSD" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "Lesser GPL and Apache License" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "MIT License" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "See license.txt" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "Slightech License" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "T.D.B" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "Version 2.0" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "WTF" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "free for academic research, for further licensing contact Wiliam Cook" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "free for research or education purpose, all rights maintained by David Applegate, William Cook, Sanjeeb Dash, and Monika Mevenkamp" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "see License.txt" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "see license.txt" cannot be mapped to valid SPDX identifier
      1 WARNING: The license value "specified in-file" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "APLv2" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Apache License, Version 2.0 (http://www.apache.org/licenses/LICENSE-2.0)" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Apache v2" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Apache v2.0" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Apache2.0" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "BSD 2-Clause License" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "BSD,GPL because of list.h; other files released under BSD,GPL" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "BSL1.0" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Binary Only" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Boost Software License 1.0" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Boost" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Check author's website" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Commercial" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "CreativeCommons-by-nc-4.0" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "ECL2.0" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Eclipse Distribution License 1.0" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "GPL because of list.h; other files released under BSD" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "GPL for sigblock" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "KHI CAD license (mesh data, see readme)" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "LGPL3" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Southwest Research Institute Proprietary" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "TERMS OF USE FOR GUNDAM RESEARCH OPEN SIMULATOR Attribution-NonCommercial-ShareAlike" cannot be mapped to valid SPDX identifier
      2 WARNING: The license value "Yujin Robot" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "ALv2" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "BSD, some icons are licensed under the GNU Lesser General Public License (LGPL) or Creative Commons Attribution-Noncommercial 3.0 License" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "BSD,LGPL,LGPL (amcl)" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "Boost Software License, Version 1.0" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "CreativeCommons-by-nc-sa-2.0" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "GPLv2 with linking exception" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "HEBI C++ Software License (https://www.hebirobotics.com/softwarelicense)" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "HOYA License" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "Mozilla Public License Version 1.1" cannot be mapped to valid SPDX identifier
      3 WARNING: The license value "N/A" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "BSD License 2.0" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "Boost Software License" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "CC-BY-SA" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "Creative Commons BY-NC-ND 3.0" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "Eclipse Public License 2.0" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "GNU General Public License v2.0" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "LGPL v2.1" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "TBD" cannot be mapped to valid SPDX identifier
      4 WARNING: The license value "zlib License" cannot be mapped to valid SPDX identifier
      5 WARNING: The license value "Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License" cannot be mapped to valid SPDX identifier
      5 WARNING: The license value "LGPL-2.1" cannot be mapped to valid SPDX identifier
      5 WARNING: The license value "Public Domain" cannot be mapped to valid SPDX identifier
      6 WARNING: The license value "BSD, GPL" cannot be mapped to valid SPDX identifier
      6 WARNING: The license value "BSD2" cannot be mapped to valid SPDX identifier
      6 WARNING: The license value "CC BY-NC-SA 4.0" cannot be mapped to valid SPDX identifier
      6 WARNING: The license value "Creative Commons" cannot be mapped to valid SPDX identifier
      6 WARNING: The license value "LGPLv2.1" cannot be mapped to valid SPDX identifier
      6 WARNING: The license value "Public domain" cannot be mapped to valid SPDX identifier
      6 WARNING: The license value "proprietary" cannot be mapped to valid SPDX identifier
      7 WARNING: The license value "BSD, Apache 2.0" cannot be mapped to valid SPDX identifier
      7 WARNING: The license value "BSD, LGPL" cannot be mapped to valid SPDX identifier
      7 WARNING: The license value "BSD,LGPL,Apache 2.0" cannot be mapped to valid SPDX identifier
      8 WARNING: The license value "GNU Lesser Public License 2.1" cannot be mapped to valid SPDX identifier
      8 WARNING: The license value "Proprietary" cannot be mapped to valid SPDX identifier
     10 WARNING: The license value "APACHE2.0" cannot be mapped to valid SPDX identifier
     10 WARNING: The license value "GNU Lesser General Public License (LGPL)" cannot be mapped to valid SPDX identifier
     14 WARNING: The license value "EPL" cannot be mapped to valid SPDX identifier
     14 WARNING: The license value "zlib" cannot be mapped to valid SPDX identifier
     16 WARNING: The license value "ASL 2.0" cannot be mapped to valid SPDX identifier
     18 WARNING: The license value "Apache" cannot be mapped to valid SPDX identifier
     20 WARNING: The license value "SwRI Proprietary" cannot be mapped to valid SPDX identifier
     20 WARNING: The license value "United States Government Purpose" cannot be mapped to valid SPDX identifier
     21 WARNING: The license value "Apache2" is not valid SPDX identifier, please use "Apache-2.0" instead
     23 WARNING: The license value "Apache License, Version 2.0" cannot be mapped to valid SPDX identifier
     26 WARNING: The license value "Apache 2" cannot be mapped to valid SPDX identifier
     31 WARNING: The license value "TODO" cannot be mapped to valid SPDX identifier
     34 WARNING: The license value "BSD-3" cannot be mapped to valid SPDX identifier
     34 WARNING: The license value "GPLv2" cannot be mapped to valid SPDX identifier
     63 WARNING: The license value "GPL" cannot be mapped to valid SPDX identifier
     73 WARNING: The license value "BSD 3-Clause" cannot be mapped to valid SPDX identifier
     75 WARNING: The license value "GPLv3" cannot be mapped to valid SPDX identifier
     77 WARNING: The license value "LGPLv3" cannot be mapped to valid SPDX identifier
     81 WARNING: The license value "LGPL" cannot be mapped to valid SPDX identifier
    742 WARNING: The license value "Apache 2.0" cannot be mapped to valid SPDX identifier
   1064 WARNING: The license value "Apache License 2.0" cannot be mapped to valid SPDX identifier
   4711 WARNING: The license value "BSD" cannot be mapped to valid SPDX identifier
shr-project commented 3 years ago

@allenh1 any comments about proposed implementation?

It's not finished yet, I want to extend the replacements to cover at least some of the values with multiple licenses listed (instead of multiple license tags), but I would appreciate your input before finishing it.