Setting up MongoDB and creating content model

[Shifat-Ali]

Changes Made

1. Added connectDB.js under config which configures mongoDB Atlas with the project
2. Added contentModel.js under models which creates contentModel as described in issue
3. Added csvSaveController.js under controllers which provides functionality to export all the documents in contentModel in database to csv file under data folder with header and name given by user
4. Added scrape.js under routes which add route for above added save to csv functionality

Extra Prerequisites

• Register on MongoDB Atlas
• Create new project IntranetSearch
• Create new database and Copy the URI to connect to it

Extra fields in .env file:

MONGODB_URI = "uri"
DB_NAME = "dbname"

Note:

• make sure to change with your password in URI
• route for save to csv [POST] is

  "/scrape/save-to-csv"

[Shifat-Ali]

Further Changes Made

1. Added dummyController.js under controllers in web-crawler to add dummy data for testing save-to-csv feature
2. Route for adding dummy data [POST] is: "/scrape/add"

Postman Collection For Testing:

Collection

[dhaneshragu]

Hey @Shifat-Ali , LGTM , but there are some changes that you have to make for this PR to be merged.

• in your error handling, I see that you have used same error handling as what I have done for elasticsearch (error?.meta?.body?.error?.reason). This wont be same for other functions, so correct this.
• In the req_filelds of your saveCSV controller, instead of giving it as a number, readability would me more if the user can provide it like ["content","url"] would be better. Make these changes, and your PR will be merged with hacktoberfest-accepted label!