search

modelfoxdotdev / modelfox

ModelFox makes it easy to train, deploy, and monitor machine learning models.
Other
1.47k stars 64 forks source link

Limit the number of tokens per column using the config file #78

Open grzegorzblaszczyk opened 2 years ago

grzegorzblaszczyk commented 2 years ago

Currently, there is no way to limit the number of tokens using the config file.

Potential config could contain JSON element like: 

"dataset": {
    "columns": [
      {
        "name": "id",
        "type": "number"
      },
      {
        "name": "author",
        "type": "text",

        ---------> "tokenLimit": 1000, <-----------------

      },
      {
        "name": "used",
        "type": "enum",
        "variants": ["true", "false"]
      }
    ]
}

so we can limit number of tokens per column.

Thanks!