Investigate on the library that can handle Data-frame like Pandas

[Softdev1]

• [x] find an better alternative that can handle dataframes in java script

• [x] In case of libraries have low community support, figure out how we can handle that internally in js/ts code

• [x] whatever the technique we have followed, that should be compatible with future integrations and langchain.js

• [x] make sure to update the related changes in existing code base by following up related issues

[s-1-n-t-h]

started investigating on danfo-js & pandas-js

will analyse it's compatability, comunity support etc and test out with a code implementaion

[s-1-n-t-h]

summary on danfojs-node:

this lib has its recent release on 10/2022, almost a year past.

while the stats show it's having a decent community support https://snyk.io/advisor/npm-package/danfojs-node and minor date realted compatability issues

Documentation is good

[s-1-n-t-h]

summary on pandas-js

https://snyk.io/advisor/npm-package/pandas-js

this package was last updated 6 yrs ago, so we can't consider this with this one stat, since our code has lot of scope to grow in coming time

[s-1-n-t-h]

the other alternative is to use Objects in js, but this can be a good idea if the data underneath is smaller in size.

as we are expecting a larger scraping in the future, this may lead to refactorings if we do it now.

danfo-js has vectorization in place that can handle & apply changes with relatively heavier data

[kesar]

i doubt you are gonna end up in an scenario where memory and objects in js is a problem

[s-1-n-t-h]

i doubt you are gonna end up in an scenario where memory and objects in js is a problem

haha, not really though! Arrays and Objects will be a good take.