search

secureIT-project / CVEfixes

CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software
Other
204 stars 52 forks source link

Parse Error When Trying to gzcat #12

Closed harik68 closed 1 year ago

harik68 commented 1 year ago

Hi, I downloaded the CVEfixes_v1.0.7.zip file from https://zenodo.org/record/7029359#.ZD2w4i-970o and unzipped to get the CVEfixes_v1.0.7.sql.gz file. But when I try to create the CVEfixes.db file, using the command given in Install.md, which is gunzip -c Data/CVEfixes_v1.0.7.sql.gz | sqlite3 Data/CVEfixes.db I am getting the following errors:

Parse error near line 142728: no such table: method_change Parse error near line 142729: no such table: method_change Parse error near line 142730: no such table: method_change Parse error near line 142731: no such table: method_change Parse error near line 142732: no such table: method_change ..........

At the end it does create the CVEfixes.db file but without the tables method_change, file_change and commits databases. How do I fix this? I am using CentOS machine for this purpose.

Thank You.

leonmoonen commented 1 year ago

Hi, Thanks for your interest in CVEfixes. I've tried to replicate your issue, but I have experienced no problems using our local copy of the same CVEfixes_v1.0.7.sql.gz as is on Zenodo. Our local files have the same md5sums as the ones reported on Zenodo, so there would be no chance if I downloaded them again.

Considering that this is such a standard rebuild of a database dump that was created by sqlite itself (nothing to do with CVEfixes really): are you sure it is not caused by memory or disk space issues on your machine? The intermediate files get rather large, and if we ignore potential on-the-fly caching, you would need at least 4G (the compressed SQL file) plus ~21GB (the uncompressed SQL file) plus 22G (final sqlite db file), and then some to keep the OS happy...

Some checksums to check your download, the content and the generated database:

md5sum CVEfixes_v1.0.7.zip
d7c90e1b3fc9b7969c7ff9ea14081145  CVEfixes_v1.0.7.zip

md5sum Data/CVEfixes_v1.0.7.sql.gz
766a1094784eba949d31b479dba1c125  Data/CVEfixes_v1.0.7.sql.gz

md5sum Data/CVEfixes.db
b18c2d55a04d3196f6ba4c83ee287e4c  Data/CVEfixes.db    # see comments below, this may not be a good check
harik68 commented 1 year ago

Hi, thanks for the prompt response. Yes, you were right, the disk space was the issue. I had only about 40GB left in the VM. When I increased the disk space (to 500GB) it worked. Now I am able to see all the tables. The checksum for CVEfixes.db is not matching though. These are what I am getting.

md5sum CVEfixes_v1.0.7.zip d7c90e1b3fc9b7969c7ff9ea14081145 CVEfixes_v1.0.7.zip

md5sum Data/CVEfixes_v1.0.7.sql.gz 766a1094784eba949d31b479dba1c125 Data/CVEfixes_v1.0.7.sql.gz

md5sum Data/CVEfixes.db 33b9aff27b4507174d84f81ed3ec36b6 Data/CVEfixes.db

The only difference I see is that I am using the command gunzip -c instead of gzcat since I couldn't install the later on the CentOS machine. Since right now I am able to use the CVEfixes.db, this is not a major issue.

Thanks and Regards, -Hari

leonmoonen commented 1 year ago

Hi Hari, Upon reading more, I may have been too quick to suggest that the md5sum of the SQLite database would be a good test. See, for example, this post https://sqlite.org/forum/forumpost/18d5f6bbaa7b3bc2?raw, which argues that similar md5sums likely mean the same information content, but also states that two databases can contain the same information while they have different internal structures and md5sums.

harik68 commented 1 year ago

Interesting. Thanks for the info.