opencybersecurityalliance / firepit

Firepit - STIX Columnar Storage
Apache License 2.0
15 stars 12 forks source link

NumericValueOutOfRange with large src_byte_count #12

Closed pcoccoli closed 3 years ago

pcoccoli commented 3 years ago

With file https://github.com/opencybersecurityalliance/stix-shifter/blob/develop/data/cybox/20000.json:

firepit cache --session test-20k 20k /home/pcoccoli/github/stix-shifter/data/cybox/20000.json
Traceback (most recent call last):
  File "/home/pcoccoli/.pyenv/versions/gh392/bin/firepit", line 33, in <module>
    sys.exit(load_entry_point('firepit', 'console_scripts', 'firepit')())
  File "/home/pcoccoli/.pyenv/versions/3.9.2/envs/gh392/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/home/pcoccoli/.pyenv/versions/3.9.2/envs/gh392/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/pcoccoli/.pyenv/versions/3.9.2/envs/gh392/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/pcoccoli/.pyenv/versions/3.9.2/envs/gh392/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/pcoccoli/.pyenv/versions/3.9.2/envs/gh392/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pcoccoli/.pyenv/versions/3.9.2/envs/gh392/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/pcoccoli/.pyenv/versions/3.9.2/envs/gh392/lib/python3.9/site-packages/typer/main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "/home/pcoccoli/github/firepit/firepit/cli.py", line 48, in cache
    db.cache(query_id, filenames)
  File "/home/pcoccoli/github/firepit/firepit/sqlstorage.py", line 246, in cache
    splitter.write(obj)
  File "/home/pcoccoli/github/firepit/firepit/splitter.py", line 220, in write
    self.writer.write_records(obj_type, self.records[obj_type], schema, self.replace, self.query_id)
  File "/home/pcoccoli/github/firepit/firepit/splitter.py", line 153, in write_records
    self.store.upsert(cursor, tablename, obj, query_id)
  File "/home/pcoccoli/github/firepit/firepit/sqlstorage.py", line 224, in upsert
    cursor.execute(stmt, values)
  File "/home/pcoccoli/.pyenv/versions/3.9.2/envs/gh392/lib/python3.9/site-packages/psycopg2/extras.py", line 236, in execute
    return super().execute(query, vars)
psycopg2.errors.NumericValueOutOfRange: integer out of range

We use type INTEGER but it looks like maybe the value "4400235700" on line 303 is too big.