process data by package

emitanaka commented 2 years ago

Should have put this here:

DanyangDai/paper-rpkg-downloads#14

emitanaka commented 2 years ago

Might need to rethink how download is extracted. It's too slow at the moment. Server sides need to process the argument so download is smaller

DanyangDai commented 2 years ago

To get the downloads for each package, it is currently downloading all the csv files. We might consider hosting a public database or hosting a private database. The function could extract the data from the database. Here are some possible platforms for us to host a database.

Monash:

ARDC Nectar Research Cloud. This is a research cloud we could use. We can access data and process them on this cloud. There are a lot of potentials from this service. Monash is partnered with Nectar so I think we could apply for this service for free. More about Monash Nectar service: Monash Nectar.
For data storing and sharing: Store.Monash. Store.Monash is a deployment of the MyTardis platform for Monash instrument facilities. The MyTardis platform assists researchers in storing, managing, sharing and disseminating data generated by scientific instruments. This is not a command line based storing place so it is easy to manage. With an appropriate license, we can make our data public with Store.Monash.

RStudio:

RStudio Cloud. The benefits for this is that it is already an R environment. We could share the project with everyone. But I am not sure about the data storing power and the processing power.
RStudio Connect. This service allow automatically updates data and generates reports. This is a great service allow people with no programming backgrounds to just read the report or dashboard. It is also very costly.

Other services:

Amazon Web Services (AWS) . This is a very powerful and convenient platform. It support many different computing and data storing and hosting service. We could use docker on AWS to process and host database.

emitanaka commented 2 years ago

Closing this as we agreed to have the processed download as package data instead. This way there shouldn't be any speed issues.

numbats / cranscrub

process data by package #1