search

dl9pf / meta-spdxscanner

SPDX scanner support
MIT License
9 stars 10 forks source link

Issues with do_spdx() on Warrior #22

Closed chaithubk closed 4 years ago

chaithubk commented 4 years ago

I am planning to use meta-spdxscanner layer to generate the SPDX docs in my poky warrior distro. I have run the fossology/fossology:3.5.0 docker image and have installed fossdriver. I am noticing this error with do_spdx()


ERROR: linux-libc-headers-5.0-r0 do_spdx: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_spdx(d)
     0003:
File: '/home/yocto/output/poky_warrior/poky/meta-spdxscanner/classes/fossdriver-host.bbclass', lineno: 137, function: do_spdx
     0133:    bb.note(' run fossdriver ...... ')
     0134:    if not os.path.isfile(tar_name):
     0135:        bb.warn(info['pn'] + "has no source, do nothing")
     0136:        return
 *** 0137:    invoke_fossdriver(tar_name,sstatefile)
     0138:    if get_cached_spdx(sstatefile) != None:
     0139:        write_cached_spdx( info,sstatefile,cur_ver_code )
     0140:        ## CREATE MANIFEST(write to outfile )
     0141:        create_manifest(info,sstatefile)
File: '/home/yocto/output/poky_warrior/poky/meta-spdxscanner/classes/fossdriver-host.bbclass', lineno: 237, function: invoke_fossdriver
     0233:    if (Reuse(server, tar_file, "Software Repository", tar_file, "Software Repository").run()  != True):
     0234:        bb.note("This OSS has not been scanned. So upload it to fossology server.")
     0235:        i = 0
     0236:        while i < 5:
 *** 0237:            if (Upload(server, tar_file, "Software Repository").run() != True):
     0238:                bb.warn("%s Upload failed, try again!" %  tar_file)
     0239:                time.sleep(delaytime)
     0240:                i += 1
     0241:            else:
File: '/home/fossdriver/fossdriver/tasks.py', lineno: 73, function: run
     0069:        logging.info("Upload complete, {} upload ID number is {}".format(self.filePath, newUploadNum))
     0070:
     0071:        # and wait until upload finishes unpacking
     0072:        logging.info("Waiting for upload {} to unpack".format(newUploadNum))
 *** 0073:        self.server.WaitUntilAgentIsDone(newUploadNum, "ununpack", pollSeconds=5)
     0074:        self.server.WaitUntilAgentIsDone(newUploadNum, "adj2nest", pollSeconds=5)
     0075:
     0076:        return True
     0077:
File: '/home/fossdriver/fossdriver/server.py', lineno: 457, function: WaitUntilAgentIsDone
     0453:            - agent: name of agent to check for.
     0454:            - pollSeconds: number of seconds to wait between polling. Defaults to 10.
     0455:        """
     0456:        # FIXME consider adding a max # of tries before returning
 *** 0457:        jobNum = self._getMostRecentAgentJobNum(uploadNum, agent)
     0458:        while not self._isJobDoneYet(jobNum):
     0459:            time.sleep(pollSeconds)
File: '/home/fossdriver/fossdriver/server.py', lineno: 297, function: _getMostRecentAgentJobNum
     0293:            - agent: name of agent to check for.
     0294:        Returns job ID number or -1 if not found.
     0295:        """
     0296:        # FIXME given _getJobsForUpload, currently retrieves just first page
 *** 0297:        jobs = self._getJobsForUpload(uploadNum)
     0298:        if jobs is None or jobs == []:
     0299:            return -1
     0300:        # will be returned in reverse chrono order, so we can just loop through
     0301:        # and stop on the first one we come to
File: '/home/fossdriver/fossdriver/server.py', lineno: 280, function: _getJobsForUpload
     0276:        # response format changed from XML to JSON on or around 3.5.0
     0277:        # see https://github.com/fossology/fossdriver/issues/17
     0278:        if self.IsAtLeastVersion("3.5.0"):
     0279:            # parse json
 *** 0280:            jobData = fossdriver.parser.parseJSONShowJobsData(results.content)
     0281:            return jobData
     0282:        else:
     0283:            # decode and parse XML
     0284:            decodedContent = fossdriver.parser.decodeAjaxShowJobsData(results.content)
File: '/home/fossdriver/fossdriver/parser.py', lineno: 209, function: parseJSONShowJobsData
     0205:    jobData = []
     0206:    js = json.loads(content.decode('utf-8'))
     0207:    showJobsData = js.get("showJobsData", [])
     0208:    for sjd in showJobsData:
 *** 0209:        jd = sjd.get("job", None)
     0210:        if jd is not None:
     0211:            jq = jd.get("jobQueue", None)
     0212:            if jq is not None:
     0213:                for k, v in jq.items():
Exception: AttributeError: 'str' object has no attribute 'get'

ERROR: linux-libc-headers-5.0-r0 do_spdx: 'str' object has no attribute 'get'
ERROR: linux-libc-headers-5.0-r0 do_spdx: Function failed: do_spdx
ERROR: Logfile of failure stored in: /home/yocto/output/poky_warrior/build/tmp/work/armv7vet2hf-neon-poky-linux-gnueabi/linux-libc-headers/5.0-r0/temp/log.do_spdx.27653
ERROR: Task (/home/yocto/output/poky_warrior/poky/meta/recipes-kernel/linux-libc-headers/linux-libc-headers_5.0.bb:do_spdx) failed with exit code '1'

@dl9pf @leimaohui Can you please help me to solve this. ?

chaithubk commented 4 years ago

To just make sure it's not due to old docker version & Warrior branch. I have tried the same on the Dunfell branch. Atleast, I see that there are no errors but the build was stuck in the do_spdx stage for a quite long time (bitbake configuration below) 

Build Configuration:
BB_VERSION           = "1.46.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "arm-poky-linux-gnueabi"
MACHINE              = "qemuarm"
DISTRO               = "poky"
DISTRO_VERSION       = "3.1"
TUNE_FEATURES        = "arm armv7ve vfp thumb neon callconvention-hard"
TARGET_FPU           = "hard"
meta                 
meta-poky            
meta-yocto-bsp       = "dunfell:a44b8d2856a937ca3991cbf566788b0cd541d777"
meta-oe              
meta-python          = "dunfell:e413c1ef621688e69bb7830bb3151ed23b30b73e"
meta-spdxscanner     = "master:c731a5f1cc83d9d8a1e120207746c89059d5576f"

Initialising tasks: 100% |###################################################################################################################################################################| Time: 0:00:10
Sstate summary: Wanted 1526 Found 12 Missed 1514 Current 1065 (0% match, 41% complete)
NOTE: Executing Tasks
WARNING: linux-libc-headers-5.4-r0 do_spdx: Creating /home/yocto/output/build/tmp/work/armv7vet2hf-neon-poky-linux-gnueabi/linux-libc-headers/5.4-r0/spdx_sstate_dir/arm-poky-linux-gnueabi/linux-libc-headers-5.4-r0/linux-libc-headers-5.4-r0-patched.tar.gz
Currently  1 running tasks (444 of 6684)   6% |##########                                                                                                                                                  |
0: linux-libc-headers-5.4-r0 do_spdx - 37m28s (pid 10172)

And I see the below errors in the fossology server running locally via docker run. 

[Sat Jun 06 10:30:43.160030 2020] [:error] [pid 237] [client 172.17.0.1:48222] PHP Fatal error:  Uncaught InvalidArgumentException: Cannot redirect to an empty URL. in /usr/local/share/fossology/vendor/symfony/http-foundation/RedirectResponse.php:86\nStack trace:\n#0 /usr/local/share/fossology/vendor/symfony/http-foundation/RedirectResponse.php(39): Symfony\\Component\\HttpFoundation\\RedirectResponse->setTargetUrl('')\n#1 /usr/local/share/fossology/www/ui/core-auth.php(219): Symfony\\Component\\HttpFoundation\\RedirectResponse->__construct('')\n#2 /usr/local/share/fossology/lib/php/Plugin/FO_Plugin.php(381): core_auth->Output()\n#3 /usr/local/share/fossology/lib/php/Plugin/FO_Plugin.php(465): FO_Plugin->getResponse()\n#4 /usr/local/share/fossology/www/ui/index.php(64): FO_Plugin->execute()\n#5 {main}\n  thrown in /usr/local/share/fossology/vendor/symfony/http-foundation/RedirectResponse.php on line 86
172.17.0.1 - - [06/Jun/2020:10:30:43 +0000] "POST /repo/?mod=auth HTTP/1.1" 500 2588 "-" "python-requests/2.20.0"
172.17.0.1 - - [06/Jun/2020:10:30:43 +0000] "GET /repo/ HTTP/1.1" 200 4820 "-" "python-requests/2.20.0"
172.17.0.1 - - [06/Jun/2020:10:30:43 +0000] "GET /repo/?mod=upload_file HTTP/1.1" 200 6483 "-" "python-requests/2.20.0"
[2020-06-06 10:30:44] Fossology\Lib\Util\DataTablesUtility.WARNING: did not find iSortingCols in inputArray [] []
172.17.0.1 - - [06/Jun/2020:10:30:43 +0000] "GET /repo/?mod=browse-processPost&folder=1&iDisplayStart=0&iDisplayLength=100 HTTP/1.1" 200 351 "-" "python-requests/2.20.0"
172.17.0.1 - - [06/Jun/2020:10:30:44 +0000] "GET /repo/?mod=upload_file HTTP/1.1" 200 6486 "-" "python-requests/2.20.0"
172.17.0.1 - - [06/Jun/2020:10:30:44 +0000] "GET /repo/?mod=upload_file HTTP/1.1" 200 6474 "-" "python-requests/2.20.0"
172.17.0.1 - - [06/Jun/2020:10:30:44 +0000] "POST /repo/?mod=upload_file HTTP/1.1" 200 6717 "http://127.0.0.1:8081/repo/?mod=upload_file" "python-requests/2.20.0"
172.17.0.1 - - [06/Jun/2020:10:30:58 +0000] "POST /repo/?mod=ajaxShowJobs&do=showjb HTTP/1.1" 200 1409 "-" "python-requests/2.20.0"
[Sat Jun 06 10:30:59.194572 2020] [:error] [pid 239] [client 172.17.0.1:48228] PHP Notice:  Undefined index: sEcho in /usr/local/share/fossology/www/ui/async/AjaxShowJobs.php on line 202
172.17.0.1 - - [06/Jun/2020:10:30:58 +0000] "GET /repo/?mod=ajaxShowJobs&do=showSingleJob&jobId=2 HTTP/1.1" 200 1293 "-" "python-requests/2.20.0"

Every 10 secs I see this error below on the fossology server:

[Sat Jun 06 11:09:03.183260 2020] [:error] [pid 179] [client 172.17.0.1:50422] PHP Notice:  Undefined index: sEcho in /usr/local/share/fossology/www/ui/async/AjaxShowJobs.php on line 202
172.17.0.1 - - [06/Jun/2020:11:09:03 +0000] "GET /repo/?mod=ajaxShowJobs&do=showSingleJob&jobId=4 HTTP/1.1" 200 26095 "-" "python-requests/2.20.0"

And the do_spdx build time keeps increasing.

I do have the fossdriver installed (shown below)

pip3 list
...
fossdriver (0.0.3, /home/fossdriver)
...

Am I doing the configuration correctly. ?

chaithubk commented 4 years ago

Closing this issue, since I have a duplicate of this same issue getting addressed here: https://github.com/fossology/fossdriver/issues/29