[TECH_TASK] File based scratch module

[Tyagi-Gaurav]

What is the problem that we're trying to solve? Create a practice module that has large input files that can serve as input.

Checklist of Tasks to do

• [x] Create large input files with multiple fields (May be CSV format?)
• [ ] Be able to read single or multiple records from files
• [x] Explore if IMDB like dataset would be useful
• [x] Document the use cases that we can code things to?

Use cases

• Read items and transform
• Read items, enhance and publish to another file
• Read items from multiple files and create a unified view.

[Tyagi-Gaurav]

Things to do.

Create separate interface for each of the above files.* Returns a Json

• [x] Transform the data and keep information in memory for quick fetch
• [x] Can we use file channels to reduce load time
• [ ] Tidy up interfaces and load all the files for the dataset
• [ ] Verify the data that it all matches.
• [ ] Verify if all titles have only single original within their dataset
• [ ] For a given movie, get all the data - name, characters, .
• [ ] For a given episode, get all the data - name, characters, seasons, etc.
• [ ] Get top 10 movies by rating of all time
• [ ] Get top 10 movies by rating in a given year
• [ ] Get all movies by an actor
• [ ] Get all movies by a director
• [ ] Get all movies common between an actor and director
• [ ] Get all common movies of a given set of actors
• [ ] Get count of number of movie names grouped by their akas

[Tyagi-Gaurav]

Reading all the items from the file is quick, but transforming and storing in map slows it down. Have verified that reading the file is not slow and all lines are read quickly.

[Tyagi-Gaurav]

Reading files using inputStream with block size of 2MB and then using in-memory manipulation of byte array, keep titles along with their file pointers works very well. Read time reduces to 40 seconds.

[Tyagi-Gaurav]

Using file channels and Byte Buffer did not change the load time for the files.