Persistant storage

[BenjaminTanguay]

Description

As a HR employee at IBM Consulting, I want to have access to a persistent list of potential candidates so that I can retrieve them at any given time

Risk

Low

Value

Medium

Story points

8

General Description

Explore database options, implements .env variable. Explore ORMs.

We need to include an ORM to streamline the database creation process. Once the database schema has been created, put it in a docker container.

Dev. shall be able to use the microservices architecture in its environment.

How

• [x] Install DotEnv package to handle .env files and variables, refactor to send token into those files @Winterhart Ideal Hours: 2h
• [x] Confirm our choice of MongoDB by doing small (MongoDB vs. SQL) @Winterhart @ddicorpo @philbeaudry @MewtR Ideal Hours: 4h
• [x] Explore technologies for the ORM (try it at home)@Winterhart @ddicorpo @philbeaudry @MewtR Ideal Hours: 4h
• [x] Build Connection Factory (allowing to use .env variable) @Winterhart @ddicorpo @philbeaudry Ideal Hours: 3h
• [x] Design & Create a data model (Schema) @Winterhart Ideal Hours: 3h
• [x] Spot recurent transaction and write //TODO: refactor-optimization @Winterhart Ideal Hours: 1h
• [x] Create a model usable by class and independent of the ORM @Winterhart Ideal Hours: 3h
• [x] Complete the TODO classes @ddicorpo @23jams @rena-21 @Winterhart @philbeaudry @MewtR
  Ideal Hours: 4h
• [x] Link docker images or Setup a Mongo Atlas DB for PROD @MewtR
  Ideal Hours: 4h
• [x] Create a docker image running the database @MewtR Ideal Hours:
• [x] Update the pipeline to include the database @MewtR Ideal Hours: 2h
• [x] Create a generic way to store/retrieve (CRUD) objects to the database (The idea here is to make generic classes that allow us to switch db tech. without to much penality IStore and Adapter pattern) Ideal Hours: 8h @Winterhart
• [x] Build Data Model Document @ddicorpo Ideal Hours: 4h

Acceptance Criteria

• [x] We have unit tests for the Interfaces
• [x] We have logging on database
• [x] CRUD operations available
• [x] Database Schema is changeable (with ORM)
• [x] It follows the microservices arch
• [x] User can develop with the database in their own environment

Acceptance Tests

(Initial) Data Model Document

[Winterhart]

We may be able to use our typescript classes with https://libraries.io/npm/typegoose

[Winterhart]

A great mongo video to start: https://www.youtube.com/watch?v=WDrU305J1yw

[Winterhart]

Hi, before going further into a specific technology. We must read/acknowledge strengths of each technology.

Please have a look at: https://www.quora.com/Which-database-you-recommend-for-web-based-big-data-reporting-with-high-speed-processing

[philbeaudry]

It seems like since MongoDB is a NoSQL, the transaction cost may be high if we have a lot of information to go through.

[philbeaudry]

Also found this: https://tutorialedge.net/typescript/typescript-mongodb-beginners-tutorial/ A good way to get started using Docker.

[Winterhart]

Watch out before creating container or files we have to agree on a tech. to use

[philbeaudry]

A free cloud MongoDB can be used, atlas: https://www.mongodb.com/cloud/atlas

• We are trying with AWS

[Winterhart]

Alright, we now have a connection factory that doesn't need .env variable (connection URL configurable on the go).

@ddicorpo and @philbeaudry You are assigned to the next task schema

Why you always want to use testingContext -> true when not running node. https://www.quora.com/Is-it-necessary-to-close-the-mongoose-connection

[Winterhart]

Database final push

In order to facilitate the completion of the story I've made a sample case with the data User. As in the data model on draw.io the schema has all field needed.

I've also design a architecture wich will allow us to be independent from the ORM technologies. The User persistant storage is made of:

• UserTDG : Insert/Update/Delete
• UserFinder: Possible find method needed
• IUserModel an interface for the field
• UserSchema a schema of the table for the ORM
• UserIntegration.test a test showing the user action

We have 8 other tables todo. I will not be able to it all. You should choose a table and start doing it.

Test your code

You should test your code with npm run test. You can also check the database for your data you can simply use the Mongo plugin in Intellij Ultimate (I've made a tutorial on it in README)

Reset Database

We still have to build the reset database command/util tool.

Remaining Table TODO

(assigned: @MewtR , @23jams , @ddicorpo , @Winterhart , @rena-21, @philbeaudry )

• User: @Winterhart => Done !!!
• Applicant : @MewtR => Done
• Token:@rena-21
• IGit:@rena-21
• IGitData : @MewtR => Done
• GitProjectSummary : @MewtR => Done
• GitProjectInput : @MewtR => Done
• HR : @Winterhart => Done

Write your name in front of at least two tables, then you can copy/paste from the User example.

[Winterhart]

Handling Business Logic

In order to make our data independent from our business logic. We will add our method/business logic in a mapper folder in domain

e.g. We will have a UserMapper with methods register login , hashPassword

src/domain/mapper/UserMapper.ts

Example of registration function

public Register(newUser: IUserModel) : Promise<boolean> {
...
// If user already created
let userExists: boolean = userFinder.findByUsername(newUser);

// Hash Password
...

// User Register

userTDG.create(newUser);

...
} 

Conclusion: Our front-end, will be able to call .../register as a POST with username, password, firstName, LastName...

[Winterhart]

The delete method in baseTDG is not done yet

[Winterhart]

baseTDG , update and delete are completed...

[Winterhart]

We having a problem with using the ref option in the mongoose...

[Winterhart]

The problem with ref has been resolved

[Winterhart]

Remaining Task

We have a few remaining tasks:

• Link storage with internal operations
• Finish the Token and Git
• Mapper for data

Remaining Questions

• Do we want to implement (create a new issue) to have API key to prevent anyone from calling the API ?
• Do we want to create route for each data class?
• What operation our current front-end needs rigth now?