Why BETS is so much slower than similar packages?

[GitHunter0]

Hey @GreedBlink , @luccasr73 . I really like BETS but one big issue that I noted is that it is around 10 times slower than other packages like wilsonfreitas/rbcb , Quandl, fredr, ...

Taking this simple example for comparison, while rbcb takes 0.25 seconds, BETS is taking 30 seconds:

library(tictoc)

tic()
BETS::BETSget(24369)
toc()

tic()
rbcb::get_series(24369) 
toc()

Any idea why such a difference? Thank you.

[GreedBlink]

@GitHunter0 , I'm glad to hear that you use the BETS package.

If you do not know the initial idea of ​​the package was to offer not only access to time series data, but also access to data analysis tools.

The package needs an update to its most basic structures. I am working on a new version, lighter (in terms of dependencies, features and functionalities) and faster.

[GitHunter0]

Hey @GreedBlink , a new version would be awesome, I'm looking forward to that!

[GitHunter0]

Hey @GreedBlink , I see here that BETS provides a lot of functionalities divided in 4 groups, which makes it comprehensive but possibly a little bit heavy.

Nonetheless, I suspect the speed issue is directly related to the database server, have you guys inspected that?

PS: Although BETS database might be something hard to maintain updated and optimized for external access, it provides useful metadata that no other package offers, which makes BETS a unique tool (also because it makes it possible to retrieve data from multiple Brazilian sources).

Thank you

[GreedBlink]

@GitHunter0 , you are right! The problem is the database connection.

I will migrate the database to AWS RDS and release a new development version. Another way to improve this is create a rest api for metadata with public access. What to you think? A simple request::GET() would be necessary and not DBI and RMySQL that is, less two packages and thus decreasing the BETS namespace.

[luccasr73]

Hi @GreedBlink! If you are going to continue with the idea of creating an api rest, I am interested in collaborating with the project if there is a possibility, an api with AWS Lambda would be really cool

[GitHunter0]

@GreedBlink , awesome news! I believe migrating to AWS RDS is the right call. And I think the rest api for metadata with public access is really a great idea! Thank you for the feedback

[GreedBlink]

Yeah i liked too!

I will create a new repository for the api and I would like to ask for your collaboration on this project (@GitHunter0 and @luccasr73), how about it? The RDS ideia is great , but any possibility to decrease the BETS imports list would be more important.

Wait for another contact in your e-mail !

Thank you for your interest in the BETS, @GitHunter0 and @luccasr73

[GitHunter0]

Sure! In what I can help, I will, @GreedBlink

[luccasr73]

Great! I am willing to help in any way I can!

[GreedBlink]

Hey @GitHunter0 , what your email?

[GitHunter0]

Hey @GitHunter0 , what your email?

I just sent you an email now. Let me know if you don't receive it

[arthurql]

Any news on this API ? I could help too.