ropensci / software-review

rOpenSci Software Peer Review.
290 stars 104 forks source link

oec package #217

Closed pachadotdev closed 5 years ago

pachadotdev commented 6 years ago

Summary

oec provides an easy way to obtain data from the Observatory of Economic Complexity by accessing its API.

Package: oec
Type: Package
Title: Observatory of Economic Complexity API Wrapper and Utility Program
Version: 2.8.2
Date: 2018-06-09
Authors@R: c(
  person("Mauricio", "Vargas S.", role = c("aut", "cre", "cph"), email = "oec@media.mit.edu"),
  person("Manuel", "Aristaran", role = c("ctb")),
  person("Pablo", "Paladino", role = c("ctb")),
  person("Gabriela", "Perez", role = c("ctb")),
  person(family = "UN Comtrade", role = c("dtc")),
  person(family = "MIT Media Lab", role = c("dtc")),
  person(family = "Datawheel", role = c("fnd"))
  )
URL: https://CRAN.R-project.org/package=oec
BugReports: https://github.com/pachamaltese/oec-r/issues
Description: Access The Observatory of Economic Complexity API from R to download international trade data.
License: MIT + file LICENSE
LazyData: TRUE
Depends:
  R (>= 3.2)
Imports:
  rlang,
  magrittr,
  dplyr,
  stringr,
  curl,
  purrr,
  jsonlite
RoxygenNote: 6.0.1
Suggests: 
    knitr,
    rmarkdown,
    testthat,
    covr
VignetteBuilder: knitr

https://github.com/pachamaltese/oec-r-package

databases, because the package connects to an API and does 3 or more API calls to simplify things for the final user who wants imports/exports and some metrics such as % of increase/decrease.

Non-expert users that use international trade data. This can also be targeted to intermediate/advanced users who can benefit from the speed and short syntax that this package provides.

Not at the moment (in 2 yrs this is the only one)

No.

Requirements

Confirm each of the following by checking the box. This package:

Publication options

Detail

None.

No.

I don't really know.

pachadotdev commented 6 years ago

@maelle can I have more time to implement all the suggestions? I'm working on them and the coolness shown on the feedback requires time to implement :)

maelle commented 6 years ago

Thanks for the update @pachamaltese, yes that's fine, please keep informing us and ask any question you might have. 🙂

pachadotdev commented 6 years ago

@maelle @aedobbyn @cimentadaj

Thanks a lot for the feedback. Here's a list of todos after the feedback, I shall be working on this after the commit I just made as a few parts require dedication. Now I finally have time to organize this and produce a good result. Have a good weekend!!

get_products("Gold") get_products("Animals")


- [x] An alternative to @cimentadaj's suggestion to allow users to get_countrycode using a regex match would be to allow users to supply either the `country_code` or the country in `get_data`; so `get_data("chl", "chn", 2015, "sitc")` would return the same data as `get_data("Chile", "China", 2015, "sitc")`. While this approach gives the user less leeway in the input they can supply than a regex would, I think most of the countries in country_codes$country are represented the way most people would expect, which saves some work on your end.
- [x] Include "all" as origin and destination in documentation and as default value for origin and destination so that the user can get up and running with the function without looking through country codes.
- [x] Fix join trade flows step to deal with `top_importer.x, top_importer.y` when using "all" as argument
- [x] Move `sitc` default to arguments instead of "if missing"
- [x] Use `webmockr` in tests
- [x] Use Styler (https://github.com/r-lib/styler) instead of Lint
- [x] Rename `oec_r_package.Rproj` to `oec.Rproj`
- [x] Check for non-Rmd files in `vignettes/`
- [x] Add one or two examples of `get_data` to the readme or the pkgdown site so that people don't have to find the vignette or go to the docs to get a preview of the package.
- [x] Switch `getdata` to `get_data`
- [x] Document results from API and explain a bit about RCA, monetary units, etc.
pachadotdev commented 6 years ago

@maelle I' m almost ready!! I implemented all of the suggestions except webmocker and styler, I'm doing that tonight. Thanks a lot @aedobbyn @cimentadaj, thanks to your comments the package improved a ton and now you can jsut write `get_data("CHILE", "ArGeNtiNa") and the internals will make it work :)

pachadotdev commented 6 years ago

@maelle now I only have webmockr implementation in unit tests and explaining a bit about result from API :D

maelle commented 6 years ago

@pachamaltese thanks for the update! Note that @sckott's webmockr doesn't support httr yet, only crul, so you can either switch to crul, or keep using httr and use https://github.com/nealrichardson/httptest

sckott commented 6 years ago

thanks @maelle - httr integration is operational in a branch on github. install like remotes::install_github("ropensci/vcr@httr-integration") and use it just like the crul integration. Would love any feedback. It should go to CRAN soon as we don't require any changes in httr itself

sckott commented 6 years ago

sorry, meant to ping you in my above comment @pachamaltese ^^

maelle commented 6 years ago

Thanks a ton for the clarification @sckott, and good news! 😃

pachadotdev commented 6 years ago

thanks @maelle @sckott ! @maelle now I completed all the changes except for the webmockr part which I am not sure how to implement

maelle commented 6 years ago

Ok thanks for the update @pachamaltese!

@sckott is there any guide, or any example using webmockr with httr in the wild (a toy example?)? I only know of https://ropensci.github.io/http-testing-book/

sckott commented 6 years ago

👋 @pachamaltese So I know exactly what you want to do, do you want to do mocking with webmockr where you simply match requests, but don't cache the responses? Or do you want more what vcr does where it matches requests and caches responses? vcr is a good fit for using in your test suite, whereas webmockr can be used for tests, but doesn't have to be, and you do have to cache your own responses with webmockr if you want them cached. Let me know then I can give examples

pachadotdev commented 6 years ago

Hi @sckott. I think it would be better to use vcr, match request and cache responses, specially when sometimes the server acts weird. Thanks a lot!!

sckott commented 6 years ago

Okay. httr integration is not yet on CRAN. You need github versions of vcr and webmockr. Install like remotes::install_github("ropensci/vcr@httr-integration")

The usage is no different from if you were using crul. So any examples, etc. in the docs should suffice. I just made a tiny package to demo using httr and vcr together https://github.com/sckott/catfact so you can see what the setup is like.

more in depth docs for vcr/webmockr here https://ropensci.github.io/http-testing-book/

pachadotdev commented 6 years ago

@sckott thanks a lot !!! I shall be reading that soon

pachadotdev commented 6 years ago

@maelle finally I implemented webmockr and I also used vcr and crul, so the functions changed a bit :)

maelle commented 6 years ago

@pachamaltese cool, congrats! So you're done with all comments?

pachadotdev commented 6 years ago

Yes! Thanks a lot for the useful comments. I just kept tidyeval for consistency.

Best,

On Wed, Aug 22, 2018, 1:32 AM Maëlle Salmon notifications@github.com wrote:

@pachamaltese https://github.com/pachamaltese cool, congrats! So you're done with all comments?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ropensci/onboarding/issues/217#issuecomment-414907770, or mute the thread https://github.com/notifications/unsubscribe-auth/AJn6OcfeL3sLKCZT8ktQV3KnFbNrQ5tRks5uTN7ygaJpZM4T-I-i .

maelle commented 6 years ago

@aedobbyn @cimentadaj can you please have a last look at the package? Thanks!

cimentadaj commented 6 years ago

Will do this weekend!

aedobbyn commented 6 years ago

Looking good @pachamaltese!

Here are a few notes. I hope to be able to to look through more thoroughly this weekend but in case I'm not able to I wanted to give you something. By topic, in no particular order:

get_data

get_countrycode

Other

This is looking really solid overall -- will be excited to use it very soon!

pachadotdev commented 6 years ago

WHOA !! thanks a lot @aedobbyn !!!!!!!!! I'm making changes this weekend. For now, the rationale for keeping the tidyeval was just to mimic some design decisions I like, for example those in the highcharter package by @jbkunst.

aedobbyn commented 6 years ago

No problem, glad it's helpful! And makes sense -- it doesn't affect the functionality of course and assuming rlang stays backward-compatible (which I think they will?) you should be fine if they change the tidyeval syntax.

pachadotdev commented 6 years ago

get_data

get_countrycode

Other

aedobbyn commented 6 years ago

Thanks for implementing those changes so quickly! Last bit of suggestions from me, I think:

pachadotdev commented 6 years ago

thanks a lot @aedobbyn !! I still have some pendings that I am adding to this new list btw, after all of this cool changes you suggested, can I contact you by email? it's for a related project that benefited a lot from this feedback mine is mvargas &a t& dcc uchile cl

cimentadaj commented 6 years ago

Great job @pachamaltese. I think @aedobbyn has covered most of the major problems. Here are some minor concerns.

Hope these are useful!

pachadotdev commented 6 years ago

@cimentadaj once again thanks for the feedback !! I completed all the points that Amanda listed (above I wrote a clickable list that I completed)

the server is down again, please give 1-2 days to complete Jorge's addresses?

@cimentadaj: any chance to contact you by email? mine is " m vargas a/t dcc uchile cl"

cimentadaj commented 6 years ago

Yes @pachamaltese , feel free to contact me and no worries about the 1-2 days.

pachadotdev commented 6 years ago

@aedobbyn @cimentadaj thanks a lot!! I had hectic days at the office, here are some changes that I'll try to complete ASAP

pachadotdev commented 6 years ago

@aedobbyn @cimentadaj Hi. Thanks for the feedback! there are four point I can't solve without breaking the check :S This weekend I' ll try to solve https://github.com/pachamaltese/oec/blob/master/R/get_data.R#L112-L161 as it is totally cleaner with that change that without

maelle commented 5 years ago

:wave: @pachamaltese did you get any chance to make progress on the package? When you do, please summarize your response again (using your checklist again I guess :wink:).

pachadotdev commented 5 years ago

@maelle Hi, I was not able to test a lot because the server has been down a lot actually. I updated the list to reflect the only 3 point I could not fix. Those can be more tricky actually!

maelle commented 5 years ago

@pachamaltese any update?

pachadotdev commented 5 years ago

Hi Maelle The server is just acting weird. Can we put this on hold? Yesterday I spoke to Amanda and Jorge about providing an alternative API that would be hassle-free. Best

On Tue, Oct 16, 2018, 1:25 AM Maëlle Salmon notifications@github.com wrote:

@pachamaltese https://github.com/pachamaltese any update?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ropensci/onboarding/issues/217#issuecomment-430096474, or mute the thread https://github.com/notifications/unsubscribe-auth/AJn6OSLbH9Nksd8-NGk4wq6cnnhuxbxjks5ulV_KgaJpZM4T-I-i .

maelle commented 5 years ago

Do you mean you don't trust the data provider enough with stability?

pachadotdev commented 5 years ago

Exactly, at the present time that' the problem. Last year it was different

On Tue, Oct 16, 2018, 11:56 AM Maëlle Salmon notifications@github.com wrote:

Do you mean you don't trust the data provider enough with stability?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ropensci/onboarding/issues/217#issuecomment-430270516, or mute the thread https://github.com/notifications/unsubscribe-auth/AJn6Oef7UB08vWE37dgCVnvIWxC0SVTIks5ulfOrgaJpZM4T-I-i .

maelle commented 5 years ago

Ok, too bad, will put the submission on hold, I hope you can solve the problem, good luck!

maelle commented 5 years ago

@pachamaltese :wave:, any news?

pachadotdev commented 5 years ago

Hi Maelle

Sorry about the long silence! I have a tentative PostgreSQL database and now I only have to work on the API. The good part is that both Jorge and Amanda are collaborating on this. If you also want to participate please do not hesitate to give ideas to create a better API.

Best!

—————

Mauricio Vargas Sepúlveda 帕夏 Do you like Data Science? visit pacha.hk 你爱科学数据专吗?你走pacha.hk

El jue., 6 de dic. de 2018 a la(s) 07:00, Maëlle Salmon ( notifications@github.com) escribió:

@pachamaltese https://github.com/pachamaltese 👋, any news?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ropensci/software-review/issues/217#issuecomment-444816124, or mute the thread https://github.com/notifications/unsubscribe-auth/AJn6OSeG4jl5GGT2v8Rmexs1HF1fz3Llks5u2OqsgaJpZM4T-I-i .

maelle commented 5 years ago

Cool to read! Is there a public repo we can link from here in case someone reads the thread and gets curious?

pachadotdev commented 5 years ago

sure !! I have created this organization to make my last 3 years of work fully reproducible https://github.com/tradestatistics

—————

Mauricio Vargas Sepúlveda 帕夏 Do you like Data Science? visit pacha.hk 你爱科学数据专吗?你走pacha.hk

El jue., 6 de dic. de 2018 a la(s) 10:22, Maëlle Salmon ( notifications@github.com) escribió:

Cool to read! Is there a public repo we can link from here in case someone reads the thread and gets curious?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ropensci/software-review/issues/217#issuecomment-444868633, or mute the thread https://github.com/notifications/unsubscribe-auth/AJn6ORf8VRizQyg9eFGZAlW8UEnD0ffUks5u2RoDgaJpZM4T-I-i .

pachadotdev commented 5 years ago

@maelle Hi!! Sorry for the very long silence. After all the issues with the "server is down" and whatever, I ended up creating my own project.

A new package based on oec is now available here https://github.com/tradestatistics/r-package

But in broader terms, tradestatistics.io is something new I made with a help from the awesome community and everything is pure R (even the API, which is where the new tradestatistics package takes the data)

There are tons of changes such as the ability to inspect different tables, filter by commodity code length, etc provided I made my own PSQL DB.

Probably I am using a laser beam to open a tuna can, but anyways I've tried do to something to foster reproducibility.

Do you have ideas to rename the package? tradestatistics sounds ok, but maybe tsapi is more appropiate.

Least but not last, api.tradestatistics.io/friend is a (boring?) joke I made with the API. It should be easy to complete the joke with the correct query.

maelle commented 5 years ago

:wave: @pachamaltese! Happy New Year! Congrats on the project (and fun endpoint)!

Should we close this issue? And when your new package is ready you can open a pre-submission inquiry (if it does the exact same things at the package using the broken server used to do, it should be in scope, but better asking first).

I find tradestatistics a better name because tsapi can be misleading since ts is often an abbreviation for time series.

pachadotdev commented 5 years ago

@maelle ok !! so I'm closing this and opening a new submission I'll prepare the new submission during the next hour

maelle commented 5 years ago

I'll prepare the new submission during the next hour

Maybe best open a pre-submission inquiry and wait until the package is more mature until a full submission (unless the package is mature already).