search

IBM / data-prep-kit

Open source project for data preparation of LLM application builders
https://ibm.github.io/data-prep-kit/
Apache License 2.0
138 stars 106 forks source link

Enhance Code2Parquet module to handle non-code text as well #520

Open shahrokhDaijavad opened 1 month ago

shahrokhDaijavad commented 1 month ago

Search before asking

Component

Other

Feature

The current version of Code2Parquet takes zip files made of text files that are all code. We need the ability to handle plain text (non-code) files as input as well.

Are you willing to submit a PR?

daw3rd commented 1 month ago

What extensions are you proposing be handled as text? Can we be more specific about requirements?

blublinsky commented 1 month ago

I do not think it matters. All the files inside zip are handled as text

shahrokhDaijavad commented 1 month ago

@daw3rd The requirement to "handle" general text (code and non-code) came from Nirmit last Friday when I was showing him the new root README files. He said in addition to code, pdf and html ingestion, we should have text too. So, when I discussed this with Boris, he convinced me that the best way to do this is to generalize code2parquet to handle arbitrary text and not just code text.

daw3rd commented 1 month ago

That does not provide any more detail on the requirements.

  1. Do you want to import .txt files from zips similarly to how we do .pdf contained in zip?
  2. Are there other extensions?
  3. In the case of code2parquet, we know it is code (.py, .java, .c, etc.). how do we know which we're importing? Or maybe we don't put any code-specific columns?

Why not merge all html, text, pdf into the same module. Probably because of the varying requirements, but to not have text2parquet, when we already have pdf2parquet and html2parquet, breaks the pattern. I'm not convinced adapting code2parquet is the right way to go.

shahrokhDaijavad commented 1 month ago

Answers and comments to your questions:

  1. Yes
  2. No
  3. I don't think Boris has changed anything that the original code2parquet was doing when the code_data parameter is by default true, so the ext column has the file extension extracted from the file path (.py, .java, ...) I assume with the new code, when code_data is false, the ext column says .txt? (@blublinsky to confirm).

When I discussed this with Boris, he mentioned the alternative option of creating an independent text2parquet, but he convinced himself (and me) that it meant duplicating almost everything that the code2parquet was doing, whereas adding a new flag to that code was much more efficient.