search

canonical / ubuntu-com-security-api

The API for CVEs and USNs data.
17 stars 9 forks source link

Missing CVEs #60

Closed szilardcserey closed 2 years ago

szilardcserey commented 2 years ago

Original title: CVE API query [ https://ubuntu.com/security/cves.json ] on package "apport" with limit=5 and offset=5 for pagination, is not fully reliable, some CVEs might by omitted by the API server

Summary

According to Ubuntu Security CVE website [https://ubuntu.com/security/cve] there are 38 CVEs listed for package "apport", Status=Released, UbuntuVersion=Xenial https://ubuntu.com/security/cve?package=apport&version=xenial&status=released

Process

However when I try to fetch the list through CVE API query [ https://ubuntu.com/security/cves.json ] with batches of 5, and stepping with offset=5, then it can happen that some CVEs are omitted by the API server like "CVE-2021-32556"

for i in $(seq 0 300); do curl -s -X GET -H "Content-Type: application/json" "https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=$((${i}*5))" | python -m json.tool | grep "^ \"id|^ \"published"; sleep 1; done

Currently the workaround we try use, and seems it works, is that we decrease the "offset" with 1, so that means limit=5 and offset=4, and then eliminate duplicate CVEs (due to the smaller offset) then we observe that previously omitted CVE is now reported by the CVE API server, so now we also receive the "CVE-2021-32556" , which was previously missing when we used the correct offset of 5 (limit of 5)

for i in $(seq 0 300); do curl -s -X GET -H "Content-Type: application/json" "https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=$((${i}*4))" | python -m json.tool | grep "^ \"id"; sleep 2; done

new update (2021.11.19): customer reported that even the upmentioned workaround does not provide reliable results, even though for myself it seemed it worked, but based on the latest report from customer, it looks like even this workaround is not reliable at all

Please let us know when the fix will be ready, currently my customer is falling back on the CVE website query, however they need the CVE API-based query to be fixed as soon as possible.

mtruj013 commented 2 years ago

@szilardcserey We've allotted time to work on this in the next iteration (about 2 weeks from now)

szilardcserey commented 2 years ago

Hi, problem still persists

A.) The number of CVEs still doesn't match for package "apport" when compared to the Ubuntu Security CVE Webpage query

___CLI API query

curl -s -X GET -H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released" \
| python -m json.tool | egrep '^ {12}"id'

this yields 20 results

___Manual web page query:

https://ubuntu.com/security/cves?q=&package=apport&priority=&version=xenial&status=released

this yields 38 results

B.) Also when we perform the query on the Ubuntu Security CVE Webpage it gives inconsistent offsetting lists compared to the full query list, missing elements and repetitive elements in the offsetting lists compared to the full list

___FULL LIST 

https://ubuntu.com/security/cves?q=&package=apport&priority=&version=xenial&status=released

38 results

CVE-2021-3710   Medium  apport  Released
CVE-2021-3709   Medium  apport  Released
CVE-2021-32556  Medium  apport  Released   <=========== THIS ONE MISSING IN THE BELOW OFFSET LISTS
CVE-2021-32555  Medium  apport  Released
CVE-2021-32554  Medium  apport  Released
CVE-2021-32553  Medium  apport  Released   <=========== THIS ONE MISSING IN THE BELOW OFFSET LISTS
CVE-2021-32552  Medium  apport  Released
CVE-2021-32551  Medium  apport  Released
CVE-2021-32550  Medium  apport  Released
CVE-2021-32549  Medium  apport  Released
CVE-2021-32547  Medium  apport  Released
CVE-2021-32548  Medium  apport  Released
CVE-2021-32557  Medium  apport  Released
CVE-2021-25683  Medium  apport  Released
CVE-2021-25684  Medium  apport  Released
CVE-2021-25682  Medium  apport  Released
CVE-2020-15702  Medium  apport  Released
CVE-2020-11936  Medium  apport  Released
CVE-2020-15701  Medium  apport  Released
CVE-2020-8831   High    apport  Released

CVE-2020-8833   Medium  apport  Released
CVE-2019-11483  Medium  apport  Released
CVE-2019-11482  Medium  apport  Released
CVE-2019-15790  Medium  apport  Released
CVE-2019-11485  Medium  apport  Released
CVE-2019-11481  Low apport  Released
CVE-2019-7307   Medium  apport  Released
CVE-2018-6552   High    apport  Released
CVE-2017-14180  High    apport  Released
CVE-2017-14177  High    apport  Released
CVE-2017-10708  Medium  apport  Released
CVE-2016-9949   Medium  apport  Released
CVE-2016-9951   Medium  apport  Released
CVE-2016-9950   Medium  apport  Released
CVE-2015-1341   High    apport  Released
CVE-2015-1338   Medium  apport  Released
CVE-2015-1325   High    apport  Released
CVE-2015-1324   High    apport  Released

___LIMIT 5   OFFSET 0

https://ubuntu.com/security/cves?q=&package=apport&priority=&version=xenial&status=released&limit=5&offset=0
CVE-2021-3710   Medium  apport  Released
CVE-2021-3709   Medium  apport  Released
CVE-2021-32550  Medium  apport  Released
CVE-2021-32548  Medium  apport  Released
CVE-2021-32547  Medium  apport  Released

curl -s -X GET -H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=0" \
| python -m json.tool | egrep '^ {12}"id'

            "id": "CVE-2021-3710",
            "id": "CVE-2021-3709",
            "id": "CVE-2021-32550",
            "id": "CVE-2021-32548",
            "id": "CVE-2021-32547",

___LIMIT 5   OFFSET 5

https://ubuntu.com/security/cves?q=&package=apport&priority=&version=xenial&status=released&limit=5&offset=5

CVE-2021-32549  Medium  apport  Released
CVE-2021-32548  Medium  apport  Released   <=== REPETITION
CVE-2021-32550  Medium  apport  Released   <=== REPETITION
CVE-2021-32551  Medium  apport  Released         
CVE-2021-32552  Medium  apport  Released

curl -s -X GET -H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=5" \
| python -m json.tool | egrep '^ {12}"id'

            "id": "CVE-2021-32549",
            "id": "CVE-2021-32548",
            "id": "CVE-2021-32550",
            "id": "CVE-2021-32551",
            "id": "CVE-2021-32552",

___LIMIT 5   OFFSET 10

https://ubuntu.com/security/cves?q=&package=apport&priority=&version=xenial&status=released&limit=5&offset=10
CVE-2021-32548  Medium  apport  Released   <=== REPETITION
CVE-2021-32547  Medium  apport  Released   <=== REPETITION
CVE-2021-32551  Medium  apport  Released   <=== REPETITION
CVE-2021-25682  Medium  apport  Released
CVE-2021-25683  Medium  apport  Released

curl -s -X GET -H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=10" \
| python -m json.tool | egrep '^ {12}"id'

            "id": "CVE-2021-32548",
            "id": "CVE-2021-32547",
            "id": "CVE-2021-32551",
            "id": "CVE-2021-25682",
            "id": "CVE-2021-25683",
___LIMIT 5   OFFSET 15

https://ubuntu.com/security/cves?q=&package=apport&priority=&version=xenial&status=released&limit=5&offset=15

CVE-2021-25682  Medium  apport  Released   <=== REPETITION
CVE-2020-15702  Medium  apport  Released
CVE-2020-11936  Medium  apport  Released
CVE-2020-15701  Medium  apport  Released
CVE-2020-8831   High    apport  Released

curl -s -X GET -H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=15" \
| python -m json.tool | egrep '^ {12}"id'

            "id": "CVE-2021-25682",
            "id": "CVE-2020-15702",
            "id": "CVE-2020-11936",
            "id": "CVE-2020-15701",
            "id": "CVE-2020-8831",
___LIMIT 5   OFFSET 20

https://ubuntu.com/security/cves?q=&package=apport&priority=&version=xenial&status=released&limit=5&offset=20

CVE-2020-8833   Medium  apport  Released   <=== REPETITION
CVE-2019-11483  Medium  apport  Released
CVE-2019-11482  Medium  apport  Released
CVE-2019-15790  Medium  apport  Released
CVE-2019-11485  Medium  apport  Released

curl -s -X GET -H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&limit=5&offset=20" \
| python -m json.tool | egrep '^ {12}"id'

            "id": "CVE-2020-8833",
            "id": "CVE-2019-11483",
            "id": "CVE-2019-11482",
            "id": "CVE-2019-15790",
            "id": "CVE-2019-11485",

Kind regards, Szilard

albertkol commented 2 years ago

Hi @szilardcserey. I pushed an API update earlier today. It may or may not have helped the missing CVE situation. Hope we can find out together.

I tried the following: https://ubuntu.com/security/cves?package=apport&version=xenial&status=released I can indeed see 38 results, as you stated.

I tried the API with the similar parameters: https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released I can find again 38 results. And I doing a Ctrl+F in my browser I can see the CVE-2021-32556 cve, it's on position 30 (offset=30&limit=1)

Could you please confirm if the problem is still there or not.

I have not tried your script. If it's necessary, I can take a look at the offset, duplication problem you are describing.

szilardcserey commented 2 years ago

Hi @albertkol

Some kind of sorting should be implemented, because the paginated results are still incoherent For example I run this (basically same) command 2 times (the only difference between the 2 is that one contains "-connect-timeout 600 -m 600 " and other does not.

And I can still see 1 CVE which is present in the first result and not present in the second, and vice-versa

curl \
-s \
-X GET \
--connect-timeout 600 \
-m 600 \
-H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=linux-hwe-5.4&status=released&version=bionic&limit=20&offset=20" \
| python -m json.tool \
| egrep '^ {12}"id'

            "id": "CVE-2022-0322",      <==== missing from the second list
            "id": "CVE-2021-4149",
            "id": "CVE-2022-27666",
            "id": "CVE-2022-1011",
            "id": "CVE-2021-45868",
            "id": "CVE-2021-39698",
            "id": "CVE-2021-39686",
            "id": "CVE-2022-27223",
            "id": "CVE-2022-26966",
            "id": "CVE-2021-26401",
            "id": "CVE-2022-0516",
            "id": "CVE-2022-23041",
            "id": "CVE-2022-23038",
            "id": "CVE-2022-23036",
            "id": "CVE-2022-23039",
            "id": "CVE-2022-23042",
            "id": "CVE-2022-23040",
            "id": "CVE-2022-23037",
            "id": "CVE-2022-0001",
            "id": "CVE-2022-23960",

curl \
-s \
-X GET \
-H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=linux-hwe-5.4&status=released&version=bionic&limit=20&offset=20" \
| python -m json.tool \
| egrep '^ {12}"id'

            "id": "CVE-2021-4203",    <====== missing from the first list
            "id": "CVE-2021-4149",
            "id": "CVE-2022-27666",
            "id": "CVE-2022-1011",
            "id": "CVE-2021-45868",
            "id": "CVE-2021-39686",
            "id": "CVE-2021-39698",
            "id": "CVE-2022-27223",
            "id": "CVE-2022-26966",
            "id": "CVE-2021-26401",
            "id": "CVE-2022-0516",
            "id": "CVE-2022-23037",
            "id": "CVE-2022-23040",
            "id": "CVE-2022-23042",
            "id": "CVE-2022-23039",
            "id": "CVE-2022-23036",
            "id": "CVE-2022-23038",
            "id": "CVE-2022-23041",
            "id": "CVE-2022-0001",
            "id": "CVE-2022-23960",

Thanks, Szilard

szilardcserey commented 2 years ago

@albertkol

Here's another example

and you can see that order of CVEs are not the same between the 2 (FIRST and SECOND) paginated queries

__FIRST PAGINATED QUERY:


curl \
-s \
-X GET \
-H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&offset=0" \
| python -m json.tool \
| egrep '^ {12}"id'

            "id": "CVE-2021-3709",
            "id": "CVE-2021-3710",
            "id": "CVE-2021-32553",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32552",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32554",
            "id": "CVE-2021-32547",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32550",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32549",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32548",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32557",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32556",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32555",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-32551",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-25684",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-25682",      <===== located at different place in 2'nd query 
            "id": "CVE-2021-25683",      <===== located at different place in 2'nd query 
            "id": "CVE-2020-11936",
            "id": "CVE-2020-15702",
            "id": "CVE-2020-15701",
            "id": "CVE-2020-8833",      <===== located at different place in 2'nd query 

curl \
-s \
-X GET \
-H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&offset=20" \
| python -m json.tool \
| egrep '^ {12}"id'

            "id": "CVE-2020-8831",
            "id": "CVE-2019-11485",      <===== located at different place in 2'nd query 
            "id": "CVE-2019-15790",      <===== located at different place in 2'nd query 
            "id": "CVE-2019-11482",      <===== located at different place in 2'nd query 
            "id": "CVE-2019-11483",      <===== located at different place in 2'nd query 
            "id": "CVE-2019-11481",      <===== located at different place in 2'nd query 
            "id": "CVE-2019-7307",
            "id": "CVE-2018-6552",
            "id": "CVE-2017-14180",
            "id": "CVE-2017-14177",
            "id": "CVE-2017-10708",
            "id": "CVE-2016-9951",      <===== located at different place in 2'nd query 
            "id": "CVE-2016-9950",      <===== located at different place in 2'nd query 
            "id": "CVE-2016-9949",      <===== located at different place in 2'nd query 
            "id": "CVE-2015-1341",
            "id": "CVE-2015-1338",
            "id": "CVE-2015-1325",
            "id": "CVE-2015-1324",

_____SECOND PAGINATED QUERY


curl \
-s \
-X GET \
-H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&offset=0" \
| python -m json.tool \
| egrep '^ {12}"id'

            "id": "CVE-2021-3709",
            "id": "CVE-2021-3710",
            "id": "CVE-2021-32556",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32555",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32554",
            "id": "CVE-2021-32557",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32552",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32551",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32550",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32549",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32548",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32547",     <===== located at different place in 1'st query 
            "id": "CVE-2021-32553",     <===== located at different place in 1'st query 
            "id": "CVE-2021-25682",     <===== located at different place in 1'st query 
            "id": "CVE-2021-25683",     <===== located at different place in 1'st query 
            "id": "CVE-2021-25684",     <===== located at different place in 1'st query 
            "id": "CVE-2020-11936",
            "id": "CVE-2020-15702",
            "id": "CVE-2020-15701",
            "id": "CVE-2020-8831",     <===== located at different place in 1'st query 

curl \
-s \
-X GET \
-H "Content-Type: application/json" \
"https://ubuntu.com/security/cves.json?package=apport&version=xenial&status=released&offset=20" \
| python -m json.tool \
| egrep '^ {12}"id'

            "id": "CVE-2020-8831",
            "id": "CVE-2019-11482",     <===== located at different place in 1'st query 
            "id": "CVE-2019-11483",     <===== located at different place in 1'st query 
            "id": "CVE-2019-11485",     <===== located at different place in 1'st query 
            "id": "CVE-2019-15790",     <===== located at different place in 1'st query 
            "id": "CVE-2019-11481",     <===== located at different place in 1'st query 
            "id": "CVE-2019-7307",
            "id": "CVE-2018-6552",
            "id": "CVE-2017-14180",
            "id": "CVE-2017-14177",
            "id": "CVE-2017-10708",
            "id": "CVE-2016-9949",     <===== located at different place in 1'st query 
            "id": "CVE-2016-9951",     <===== located at different place in 1'st query 
            "id": "CVE-2016-9950",     <===== located at different place in 1'st query 
            "id": "CVE-2015-1341",
            "id": "CVE-2015-1338",
            "id": "CVE-2015-1325",
            "id": "CVE-2015-1324",
albertkol commented 2 years ago

@szilardcserey I think the switching of positions happens when the published date is the same. So I added to the sorting a second criteria by id. I hope this sorts out the problem. (see what I did there?)

Could you confirm if the problems highlighted in this issues are solved? We'd like to close this issue. But happy to continue the conversation in new issues.

szilardcserey commented 2 years ago

@albertkol thanks for adding sorting functionality, I have tested and the results are pretty solid right now, I asked my customer as well to test it. So I think you can close this case. Many thanks for your proactive and quick help!