search

fasten-project / vulnerability-producer

Gathers, enriches and publishes vulnerability information to a Kafka topic.
https://www.fasten-project.eu/
Apache License 2.0
6 stars 3 forks source link

Producer crashes when parsing oss-fuzz-vulns, then restarts from scratch #90

Closed MagielBruntink closed 2 years ago

MagielBruntink commented 3 years ago

This is with the latest master version, ie. the fasten.security image, version 21c55073.

Crash log:

[2021-06-29 15:44:28,816] [INFO ] [main] [e.f.v.u.p.ParserManager] - Gathering vulnerabilities from oss-fuzz-vulns
[2021-06-29 15:44:28,854] [ERROR] [main] [e.f.v.u.c.GitHelper] - Could not clone https://github.com/google/oss-fuzz-vulns.git. Maybe repo already cloned locally.

Cannot create property=ecosystem_specific for JavaBean=eu.fasten.vulnerabilityproducer.utils.mappers.YAMLHandler$OSSFuzzMapper@54bbe00e
 in 'string', line 1, column 1:
    id: OSV-2020-969
    ^
Unable to find property 'ecosystem_specific' on class: eu.fasten.vulnerabilityproducer.utils.mappers.YAMLHandler$OSSFuzzMapper
 in 'string', line 25, column 3:
      severity: HIGH
      ^

    at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:292)
    at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:171)
    at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:331)
    at org.yaml.snakeyaml.constructor.BaseConstructor.constructObjectNoCheck(BaseConstructor.java:230)
    at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:219)
    at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:173)
    at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:157)
    at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:472)
    at org.yaml.snakeyaml.Yaml.load(Yaml.java:398)
    at eu.fasten.vulnerabilityproducer.utils.parsers.OSSFuzzParser.getVulnerabilities(OSSFuzzParser.java:71)
    at eu.fasten.vulnerabilityproducer.utils.parsers.ParserManager.getVulnerabilitiesFromParsers(ParserManager.java:99)
    at eu.fasten.vulnerabilityproducer.VulnerabilityProducer.start(VulnerabilityProducer.java:111)
    at eu.fasten.vulnerabilityproducer.Main.run(Main.java:134)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1729)
    at picocli.CommandLine.access$900(CommandLine.java:145)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2101)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2068)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
    at picocli.CommandLine.execute(CommandLine.java:1864)
    at eu.fasten.vulnerabilityproducer.Main.main(Main.java:82)
Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find property 'ecosystem_specific' on class: eu.fasten.vulnerabilityproducer.utils.mappers.YAMLHandler$OSSFuzzMapper
    at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:159)
    at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:148)
    at org.yaml.snakeyaml.TypeDescription.discoverProperty(TypeDescription.java:254)
    at org.yaml.snakeyaml.TypeDescription.getProperty(TypeDescription.java:265)
    at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:231)
    ... 19 more
MagielBruntink commented 3 years ago

Similar issue with the latest develop image:

[2021-06-29 16:04:03,722] [INFO ] [main] [e.f.v.u.c.GitHelper] - Cloned https://github.com/google/oss-fuzz-vulns.git
Cannot create property=database_specific for JavaBean=eu.fasten.vulnerabilityproducer.utils.mappers.YAMLHandler$OSSFuzzMapper@13bd4a20
 in 'string', line 1, column 1:
    id: OSV-2020-969
    ^
Unable to find property 'database_specific' on class: eu.fasten.vulnerabilityproducer.utils.mappers.YAMLHandler$OSSFuzzMapper
 in 'string', line 27, column 3:
      introduced_range: unknown:c4dcac ... 
      ^

    at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:292)
    at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.construct(Constructor.java:171)
    at org.yaml.snakeyaml.constructor.Constructor$ConstructYamlObject.construct(Constructor.java:331)
    at org.yaml.snakeyaml.constructor.BaseConstructor.constructObjectNoCheck(BaseConstructor.java:230)
    at org.yaml.snakeyaml.constructor.BaseConstructor.constructObject(BaseConstructor.java:219)
    at org.yaml.snakeyaml.constructor.BaseConstructor.constructDocument(BaseConstructor.java:173)
    at org.yaml.snakeyaml.constructor.BaseConstructor.getSingleData(BaseConstructor.java:157)
    at org.yaml.snakeyaml.Yaml.loadFromReader(Yaml.java:472)
    at org.yaml.snakeyaml.Yaml.load(Yaml.java:398)
    at eu.fasten.vulnerabilityproducer.utils.parsers.OSSFuzzParser.getVulnerabilities(OSSFuzzParser.java:71)
    at eu.fasten.vulnerabilityproducer.utils.parsers.ParserManager.getVulnerabilitiesFromParsers(ParserManager.java:99)
    at eu.fasten.vulnerabilityproducer.VulnerabilityProducer.start(VulnerabilityProducer.java:111)
    at eu.fasten.vulnerabilityproducer.Main.run(Main.java:134)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1729)
    at picocli.CommandLine.access$900(CommandLine.java:145)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2101)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2068)
    at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1935)
    at picocli.CommandLine.execute(CommandLine.java:1864)
    at eu.fasten.vulnerabilityproducer.Main.main(Main.java:82)
Caused by: org.yaml.snakeyaml.error.YAMLException: Unable to find property 'database_specific' on class: eu.fasten.vulnerabilityproducer.utils.mappers.YAMLHandler$OSSFuzzMapper
    at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:159)
    at org.yaml.snakeyaml.introspector.PropertyUtils.getProperty(PropertyUtils.java:148)
    at org.yaml.snakeyaml.TypeDescription.discoverProperty(TypeDescription.java:254)
    at org.yaml.snakeyaml.TypeDescription.getProperty(TypeDescription.java:265)
    at org.yaml.snakeyaml.constructor.Constructor$ConstructMapping.constructJavaBean2ndStep(Constructor.java:231)
    ... 19 more

Seems like the input format are changing quite a lot! Could you please consider to have the producer not crash on parsing errors, but instead log error and proceed?

MagielBruntink commented 3 years ago

Should be fixed by 91b8400 for now.