No such Column as filelines error when running integrate

[Poiuy7312]

chasten integrate all-programs C:\Users\Preston\CMPSC-203\ChastenProject\chasten\subject-data\lazytracker\chasten-results-lazytracker-20230913151249-d0046ac472434a709f6489e53f28b042.json -s subject-data/

The following error appears when I run the above command

💫 chasten: Analyze the AST of Python Source Code
🔗 GitHub: https://github.com/gkapfham/chasten

✨ Combining data file(s) in:

• Directory: C:\Users\Preston\CMPSC-203\ChastenProject\chasten\subject-data\lazytracker
  • File: 'chasten-results-lazytracker-20230913151249-d0046ac472434a709f6489e53f28b042.json'

✨ Saved the file 'chasten-integrated-results-all-programs-20230913151315-18a0214c8b664b6ea93b86a5622c375b.json'
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ C:\Users\Preston\CMPSC-203\ChastenProject\chasten\chasten\main.py:742 in integrate               │
│                                                                                                  │
│   739 │   │   output.console.print(f"\n:sparkles: Saved the file '{combined_json_file_name}'")   │
│   740 │   # "flatten" (i.e., "un-nest") the now-saved combined JSON file using flatterer         │
│   741 │   # create the SQLite3 database and then configure the database for use in datasett      │
│ ❱ 742 │   combined_flattened_directory = filesystem.write_flattened_csv_and_database(            │
│   743 │   │   combined_json_file_name,                                                           │
│   744 │   │   output_directory,                                                                  │
│   745 │   │   project,                                                                           │
│                                                                                                  │
│ C:\Users\Preston\CMPSC-203\ChastenProject\chasten\chasten\filesystem.py:280 in                   │
│ write_flattened_csv_and_database                                                                 │
│                                                                                                  │
│   277 │   # create a view that combines all of the data                                          │
│   278 │   database.create_chasten_view(database_file_name_str)                                   │
│   279 │   # enable full-text search in the SQLite3 database                                      │
│ ❱ 280 │   database.enable_full_text_search(database_file_name_str)                               │
│   281 │   # return the name of the directory that contains the flattened CSV files               │
│   282 │   return flattened_output_directory_str                                                  │
│   283                                                                                            │
│                                                                                                  │
│ C:\Users\Preston\CMPSC-203\ChastenProject\chasten\chasten\database.py:67 in                      │
│ enable_full_text_search                                                                          │
│                                                                                                  │
│    64 │   │   ]                                                                                  │
│    65 │   )                                                                                      │
│    66 │   # enable full-text search on the sources database table                                │
│ ❱  67 │   database["sources"].enable_fts(                                                        │
│    68 │   │   [                                                                                  │
│    69 │   │   │   "filename",                                                                    │
│    70 │   │   │   "filelines",                                                                   │
│                                                                                                  │
│ C:\Users\Preston\AppData\Local\pypoetry\Cache\virtualenvs\chasten-Ti8Qy4ta-py3.11\Lib\site-packa │
│ ges\sqlite_utils\db.py:2358 in enable_fts                                                        │
│                                                                                                  │
│   2355 │   │   │   self.disable_fts()                                                            │
│   2356 │   │                                                                                     │
│   2357 │   │   self.db.executescript(create_fts_sql)                                             │
│ ❱ 2358 │   │   self.populate_fts(columns)                                                        │
│   2359 │   │                                                                                     │
│   2360 │   │   if create_triggers:                                                               │
│   2361 │   │   │   old_cols = ", ".join("old.[{}]".format(c) for c in columns)                   │
│                                                                                                  │
│ C:\Users\Preston\AppData\Local\pypoetry\Cache\virtualenvs\chasten-Ti8Qy4ta-py3.11\Lib\site-packa │
│ ges\sqlite_utils\db.py:2408 in populate_fts                                                      │
│                                                                                                  │
│   2405 │   │   │   │   table=self.name, columns=", ".join("[{}]".format(c) for c in columns)     │
│   2406 │   │   │   )                                                                             │
│   2407 │   │   )                                                                                 │
│ ❱ 2408 │   │   self.db.executescript(sql)                                                        │
│   2409 │   │   return self                                                                       │
│   2410 │                                                                                         │
│   2411 │   def disable_fts(self) -> "Table":                                                     │
│                                                                                                  │
│ C:\Users\Preston\AppData\Local\pypoetry\Cache\virtualenvs\chasten-Ti8Qy4ta-py3.11\Lib\site-packa │
│ ges\sqlite_utils\db.py:524 in executescript                                                      │
│                                                                                                  │
│    521 │   │   """                                                                               │
│    522 │   │   if self._tracer:                                                                  │
│    523 │   │   │   self._tracer(sql, None)                                                       │
│ ❱  524 │   │   return self.conn.executescript(sql)                                               │
│    525 │                                                                                         │
│    526 │   def table(self, table_name: str, **kwargs) -> Union["Table", "View"]:                 │
│    527 │   │   """                                                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OperationalError: no such column: filelines

The only way I've been able to stop this error from occurring so far is by removing "filelines" from the following code in database.py

database["sources"].enable_fts(
        [
            "filename",
            "filelines",
            "check_id",
            "check_name",
            "check_description",
            "check_pattern",
        ]
    )

Which then results in integrate running with no errors

💫 chasten: Analyze the AST of Python Source Code
🔗 GitHub: https://github.com/gkapfham/chasten

✨ Combining data file(s) in:

• Directory: C:\Users\Preston\CMPSC-203\ChastenProject\chasten\subject-data\lazytracker
  • File: 'chasten-results-lazytracker-20230913151249-d0046ac472434a709f6489e53f28b042.json'

✨ Saved the file 'chasten-integrated-results-all-programs-20230913151430-f51221df497746a19191baa735dc6087.json'

✨ Created this directory structure in C:\Users\Preston\CMPSC-203\ChastenProject\chasten\subject-data:

📂 chasten-flattened-csvs-sqlite-db-all-programs-20230913151430-90f2041a3273407e963e8b13cb404f5a
├── 📄 chasten.db
├── 📂 csv
│   ├── 📄 main.csv
│   ├── 📄 sources.csv
│   └── 📄 sources_check_matches.csv
├── 📄 datapackage.json
├── 📄 fields.csv
└── 📄 tables.csv

[Poiuy7312]

I found this issue issue when working on #38

[gkapfham]

Hello @Poiuy7312, it looks like this issue is arising because of the fact that there is an attribute that was previously in the database table and it is now not in the table. This causes a problem with the program attempts to create an index on that attribute.

Please consider options like the following:

• Remove the attribute if you are confident that it is not needed; then remove the creation of the index in the database on this attribute
• Keep the attribute if you think that there are benefits to including it; then make sure to keep the index to it for this attribute in the database

What do you think is the best strategy?