AnswerDotAI / llms-txt

The /llms.txt file, helping language models use your website
http://llmstxt.org/
Apache License 2.0
212 stars 17 forks source link

Error when `llms.txt` contains level-2 sections and no links #16

Open gadenbuie opened 4 days ago

gadenbuie commented 4 days ago

If the llms.txt document contains a level-2 section, parse_link() throws if there aren't any links in the document. Here's an example using a variation of a test case:

cat << 'EOF' > llms.txt
# No Links Title

## Description

> No description

Some details without links
EOF

llms_txt2ctx llms.txt
Traceback (most recent call last):
  File "/Users/garrick/work/external/AnswerDotAI/llms-txt/.venv/bin/llms_txt2ctx", line 8, in <module>
    sys.exit(llms_txt2ctx())
             ~~~~~~~~~~~~^^
  File "/Users/garrick/work/external/AnswerDotAI/llms-txt/.venv/lib/python3.13/site-packages/fastcore/script.py", line 121, in _f
    return tfunc(**merge(args, args_from_prog(func, xtra)))
  File "/Users/garrick/work/external/AnswerDotAI/llms-txt/llms_txt/core.py", line 125, in llms_txt2ctx
    ctx = create_ctx(Path(fname).read_text(), optional=optional, n_workers=n_workers)
  File "/Users/garrick/work/external/AnswerDotAI/llms-txt/llms_txt/core.py", line 112, in create_ctx
    d = parse_llms_file(txt)
  File "/Users/garrick/work/external/AnswerDotAI/llms-txt/llms_txt/core.py", line 57, in parse_llms_file
    start,sects = _parse_llms(txt)
                  ~~~~~~~~~~~^^^^^
  File "/Users/garrick/work/external/AnswerDotAI/llms-txt/llms_txt/core.py", line 51, in _parse_llms
    sects = {k: _parse_links(v) for k,v in d.items()}
                ~~~~~~~~~~~~^^^
  File "/Users/garrick/work/external/AnswerDotAI/llms-txt/llms_txt/core.py", line 45, in _parse_links
    return [parse_link(l) for l in re.split(r'\n+', links.strip()) if l.strip()]
            ~~~~~~~~~~^^^
  File "/Users/garrick/work/external/AnswerDotAI/llms-txt/llms_txt/core.py", line 41, in parse_link
    return re.search(pat, txt).groupdict()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'groupdict'