aboutcode-org / scancode-toolkit

:mag: ScanCode detects licenses, copyrights, dependencies by "scanning code" ... to discover and inventory open source and third-party packages used in your code. Sponsored by NLnet project https://nlnet.nl/project/vulnerabilitydatabase, the Google Summer of Code, Azure credits, nexB and others generous sponsors!
https://github.com/aboutcode-org/scancode-toolkit/releases/
2.07k stars 536 forks source link

Failure of consolidate plugin #3134

Open NickHastings opened 1 year ago

NickHastings commented 1 year ago

Description

Running scancode with consolidate plugin failed on zig source tree.

How To Reproduce

% wget https://ziglang.org/download/0.9.1/zig-0.9.1.tar.xz 
% tar xf zig-0.9.1.tar.xz
% cd zig-0.9.1
% scancode -n 6 --license --copyright --license-text --consolidate --debian ~/debian-copyright .
Setup plugins...                                                                                                                                 
Collect file inventory...                                                                                                                        
Scan files for: licenses, copyrights with 6 process(es)...
[####################] 28456                                                                                                                     
/home/hastings/.local/lib/python3.9/site-packages/scancode/cli.py:1077: ConsolidatorPluginDeprecationWarning: The --consolidate option will be de
precated in a future version of scancode-toolkit.
  plugin.process_codebase(codebase, **kwargs)                           
ERROR: failed to run post-scan plugin: consolidate:                                                                                              
Traceback (most recent call last):                                      
  File "/home/hastings/.local/lib/python3.9/site-packages/scancode/cli.py", line 1077, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)                                                                                                  
  File "/home/hastings/.local/lib/python3.9/site-packages/summarycode/plugin_consolidate.py", line 176, in process_codebase
    consolidations.extend(get_holders_consolidated_components(codebase))                                                                         
  File "/home/hastings/.local/lib/python3.9/site-packages/summarycode/plugin_consolidate.py", line 399, in get_holders_consolidated_components
    for c in create_consolidated_components(resource, codebase, holder):                                                                         
  File "/home/hastings/.local/lib/python3.9/site-packages/summarycode/plugin_consolidate.py", line 413, in create_consolidated_components
    if not (normalized_holder.key == holder_key):                                                                                                
AttributeError: 'dict' object has no attribute 'key'                                                                                             

Scanning done.                      
Some files failed to scan properly:                                     
ERROR: failed to run post-scan plugin: consolidate:                                                                                              
Traceback (most recent call last):                                      
  File "/home/hastings/.local/lib/python3.9/site-packages/scancode/cli.py", line 1077, in run_codebase_plugins
    plugin.process_codebase(codebase, **kwargs)                                                                                                  
  File "/home/hastings/.local/lib/python3.9/site-packages/summarycode/plugin_consolidate.py", line 176, in process_codebase
    consolidations.extend(get_holders_consolidated_components(codebase))                                                                         
  File "/home/hastings/.local/lib/python3.9/site-packages/summarycode/plugin_consolidate.py", line 399, in get_holders_consolidated_components
    for c in create_consolidated_components(resource, codebase, holder):                                                                         
  File "/home/hastings/.local/lib/python3.9/site-packages/summarycode/plugin_consolidate.py", line 413, in create_consolidated_components
    if not (normalized_holder.key == holder_key):                                                                                                
AttributeError: 'dict' object has no attribute 'key'                                                                                             
Summary:        licenses, copyrights with 6 process(es)                                                                                          
Errors count:   1                   
Scan Speed:     9.98 files/sec.                                         
Initial counts: 15400 resource(s): 14228 file(s) and 1172 directorie(s)                                                                          
Final counts:   15400 resource(s): 14228 file(s) and 1172 directorie(s)                                                                          
Timings:                            
  scan_start: 2022-10-24T234607.752111                                  
  scan_end:   2022-10-25T001006.418499                                  
  setup_scan:licenses: 1.42s                                            
  setup: 1.42s                      
  inventory: 2.84s                  
  scan: 1425.52s                    
  post-scan:consolidate: 3.08s                                          
  post-scan: 3.08s                  
  output:debian: 3.70s              
  output: 3.70s                     
  total: 1443.01s                   
Removing temporary files...done.                                        
scancode -n 6 --license --copyright --license-text --consolidate --debian  .  8545.88s user 14.63s system 592% cpu 24:04.62 total

System configuration

pombredanne commented 1 year ago

@NickHastings Thanks! @JonoYang ping

JonoYang commented 1 year ago

@NickHastings Thanks for the report! I'm able to reproduce this error on my end. From an immediate glance, it looks like we're slipping a dictionary into a list of Text() objects. I'll have to dig more to see where/how that's happening. Interestingly, I do not encounter this error when I scan something else, like https://codeload.github.com/rust-lang/libc/tar.gz/refs/tags/0.2.121