db csv import - not finished

[louisevdb84]

What needs to be done here?

[atherdon]

before we'll start to work on this part, i propose you to help me with 2 other repositories, that have import, but they have first, oldest version. Will you be able take a look? https://github.com/GroceriStar/groceristar https://github.com/ChickenKyiv/recipe-api-only

[louisevdb84]

Yes, would like to. Is there a reason that Mongoose is not being used? Can I try and implement Mongoose in the import script?

[atherdon]

i assume no. maybe loopback will have some restrictions. try to google loopback csv import, but again - this should be done only after implementing updates of import script to CK and GS. Can you do it? because i assume you're ready

[louisevdb84]

Yes, would definitely like to try doing it on CK and GS.

[atherdon]

ok, it's a good news! feel free to fork them and apply your code there

[atherdon]

About your question, related to mongoose import. I assume at some time - we'll move into graphQL, so later it's better to use that approach

[louisevdb84]

Regarding my pull request https://github.com/ChickenKyiv/recipe-api-only/pull/201:

The current import script did not work on recipes. I fixed a couple of bugs and changed file names to conform with https://github.com/ChickenKyiv/recipe-search-api naming.

Please confirm whether the import scripts and the helper script in these files needs to look exactly like https://github.com/ChickenKyiv/recipe-search-api?

[atherdon]

ok, i'll take a look, but can you explain more "didn't work on recipes"? any additional information can save me a lot of time. maybe screenshots or video with import...

---

Btw, related to csv import

We can use middleware, related to LoopBack: https://www.npmjs.com/package/loopback-import-mixin https://github.com/cRicateau/loopback-csv-import https://www.theodo.fr/blog/2016/01/how-to-make-a-user-friendly-and-transactional-csv-import-in-loopback/ https://stackoverflow.com/questions/38095885/read-data-from-file-using-loopback

if we'll not been able to import via default loopback way - we'll try to connect to mongo and use mongoose http://stackoverflow.com/questions/25054958/save-a-very-big-csv-to-mongodb-using-mongoose http://stackoverflow.com/questions/30462849/import-text-files-csv-format-into-mongodb-using-mongoose-schema https://www.npmjs.com/package/mongoose-csv https://www.npmjs.com/package/mongoose-to-csv https://gist.github.com/killercup/11256050 https://gist.github.com/jaywon/e2067b689192f245cd7b https://github.com/wdavidw/node-csv http://thisisnotanexit.uk/uploading-data-from-csv-into-mongodb/

GroceriStar/csv-import#1

[louisevdb84]

The bugs was some minor miss typings, like missing semi colons, missing commas. And variables declared as for example Recipes2 and used as Recipe1. Department was used, but not instantiated/declared. I made comments on the changes in the pull request.

[atherdon]

got it. it was my mistakes. thanks!

[atherdon]

i think this is one of the reasons why we should try to handle that import via csv files

[atherdon]

not sure, but maybe we should have some clean repository with loopback instance - where you can play with csv import? will this be more comfortable for you? only today i realize how messy was import at recipe-api server

[louisevdb84]

ok, yes, would be easier to have a clean repository.

[atherdon]

ok, can you jump into https://github.com/ChickenKyiv/recipe-search-api? import at this repo is not so bad :) so we can test csv import here. I can prepare a file for you, that will help us also test how current endpoints work. because from February API testing of search is pending... maybe you also help me with testing? will be cool! i have a lot of data, i have test cases - but never have time at last month to finish it.

in the mean time - i'll make some cool updates in recipe-api. i start to realize how dumb is to have similar imports in different projects, and just copy-paste the code. so i'll create an npm package that we'll be able to install and use

[louisevdb84]

ok, will start looking at CSV import tomorrow.

Havent done testing before, but would like to learn, so yes, will try to help you.

Wow, sounds very cool what you want to do with recipe-api :)

[atherdon]

i noticed that you have project related to videos. I have an loopback api server for storing youtube video links - it'll be cool if you can use it in your project

[louisevdb84]

cool, will have a look at that

[louisevdb84]

I had a look at the links you sent to me regarding loopback csv import. I'm having a hard time understanding how it works. I am following this example: https://github.com/cRicateau/loopback-csv-import but they use coffee script and I'm unfamiliar with it. I cant find other examples for loopback csv import which doesnt use coffee script.

Ive tried implementing it on recipe-search-api using coffee script, but getting errors which I don't understand. Should I create a pull request so that you can have a look at it?

[atherdon]

Coffee Script is just another type of short versioning of JS. It was cool a few years ago, but then Babel with ES6 standard came to the scene and made a lot of this custom versions unpopular. People still using it, btw.

I checked the CoffeeScript version. looks hard to understand, but i can read that code. if we'll not find a solution together maybe i'll try to convert it to plain JS myself. Another option is to use a clean nidejs script, without a lot of loopback methods in order to manipulate mongo source, but i'm not completely sure that this is a perfect idea, because we can miss something.

Btw, did you check this link: https://stackoverflow.com/questions/38095885/read-data-from-file-using-loopback It looks like a very clear solution.

I also checked this link, it looks scary, but people downloading it quite often... https://www.npmjs.com/package/loopback-import-mixin

[atherdon]

Sure, please create a PR, so i can check the code and understand more

[louisevdb84]

I am trying to pull the latest changes, and get the following message:

[louisevdb84]

Never mind. I tried it again, and somehow it just worked. After I pulled from upstream master, I have merge conflicts in package files. Is there an easy way to fix it, without having to manually fix each conflict?

[atherdon]

i think it's a good question. and if you want to resolve conflicts without jumping between files - you should rollower your changes, then checkout the latest version of code from main repo and then work on the changes. i want to help you, but i think this trouble will improve your git skills :)