crocs-muni / sec-certs

Tool for analysis of security certificates and their security targets (Common Criteria, NIST FIPS140-2...).
https://sec-certs.org
MIT License
9 stars 7 forks source link

Try to fix FIPS alg search. #418

Closed J08nY closed 2 weeks ago

J08nY commented 2 weeks ago

The link: https://csrc.nist.gov/projects/Cryptographic-Algorithm-Validation-Program/validation-search?searchMode=implementation&page=1&ipp=250 does not work, gives 408. Their search on the site which queries this also does not work.

However, when one searches by "validation" it does work. There is no difference to us, as we do not have any queries and simply want all the data.

J08nY commented 2 weeks ago

ffs, now I am getting:

KeyError: 'Validation Number'
  File "pandas/core/indexes/base.py", line 3802, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 165, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 5745, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 5753, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'Validation Number'
  File "dramatiq/worker.py", line 485, in process_message
    res = actor(*message.args, **message.kwargs)
  File "dramatiq/actor.py", line 177, in __call__
    return self.fn(*args, **kwargs)
  File "sec_certs_page/common/tasks.py", line 503, in wrapper
    return f(*args, **kwargs)
  File "sec_certs_page/fips/tasks.py", line 171, in update_data
    updater.update()
  File "sec_certs_page/common/tasks.py", line 337, in update
    raise e
  File "sec_certs_page/common/tasks.py", line 253, in update
    to_reindex = self.process(dset, paths)
  File "sec_certs_page/fips/tasks.py", line 127, in process
    dset.process_auxiliary_datasets(update_json=False)
  File "sec_certs/serialization/json.py", line 108, in _serialize
    result = func(*args, **kwargs)
  File "sec_certs/dataset/fips.py", line 253, in process_auxiliary_datasets
    self.auxiliary_datasets.algorithm_dset = self._prepare_algorithm_dataset(download_fresh)
  File "sec_certs/utils/profiling.py", line 41, in wrapper
    return func(*args, **kwargs)
  File "sec_certs/dataset/fips.py", line 258, in _prepare_algorithm_dataset
    alg_dset = FIPSAlgorithmDataset.from_web(self.algorithm_dataset_path)
  File "sec_certs/dataset/fips_algorithm.py", line 58, in from_web
    algs = set(itertools.chain.from_iterable(FIPSAlgorithmDataset.parse_algorithms_from_html(x) for x in htmls))
  File "sec_certs/dataset/fips_algorithm.py", line 58, in <genexpr>
    algs = set(itertools.chain.from_iterable(FIPSAlgorithmDataset.parse_algorithms_from_html(x) for x in htmls))
  File "sec_certs/dataset/fips_algorithm.py", line 99, in parse_algorithms_from_html
    df["alg_type"] = df["Validation Number"].map(lambda x: re.sub(r"[0-9\s]", "", x))
  File "pandas/core/frame.py", line 3807, in __getitem__
    indexer = self.columns.get_loc(key)
  File "pandas/core/indexes/base.py", line 3804, in get_loc
    raise KeyError(key) from err