search

Pomax / node-flickrapi

A node.js (and client-library) implementation of the Flickr API with oauth API key authentication and API method proxying
177 stars 51 forks source link

Allow Flickr data to be written to another resource #55

Closed Wildhoney closed 8 years ago

Wildhoney commented 9 years ago

Syncing the data directory on first-run whilst running locally works just fine, but when using a PaaS such as Heroku it breaks the rules of a twelve factor app and therefore has to re-download the data directory each and every time. This can be especially problematic when your environment is read-only. Curiously enough, node-flickrapi seems to break silently whilst downloading the data to a read-only environment.

I propose that you allow the data to be written to another resource, such as Redis or MongoDB. Alternatively you could bypass the syncing of data entirely, although I'm not sure how feasible that is for this module.

Pomax commented 9 years ago

you don't need to sync anything if you don't want to, it's primarily an API node module, with the downsync simply being a bonus because that's the most immediate "I just want to ..." thing people want -- I agree that it might be useful to use an external data repository, although I'm not sure I'd be too keen on writing one mostly because the idea of using heroku or the like for a permanent mirror seems a conflict of needs. A digtial ocean droplet would make far more sense, at which point you get a regular fs again, with file permanence rather than prebuilt restarting slugs/dynos/etc

Wildhoney commented 9 years ago

How would I then configure the module to not download the API endpoint data?

Pomax commented 9 years ago

ahh, missed that part, you're right.

Wildhoney commented 9 years ago

You're not helping me much :kissing: Assuming I am using a PaaS — such as Heroku — how would I disable the downsync of the API data, and still retain the ability to use flickr.photosets.getPhotos? Currently I've committed the data directory into the repository, but that's not ideal.

Pomax commented 9 years ago

There's two things here: downloading the API information "on startup", and writing that information to file as a caching mechanism for subsequent startups. If you're running the library off of a slug-based PaaS like Heroku, where a dyno restart wipes the file system, we'd need both an external filesystem module or json store, and change the loader code to understand how to "what am I access"-agnostically read and write from/to them.

The loader code currently checks for the presence of cached files in https://github.com/Pomax/node-flickrapi/blob/master/src/flickr-api-object.js#L88, and blind (re)writes them to disk at https://github.com/Pomax/node-flickrapi/blob/master/src/flickr-api-object.js#L105, so in order to make the loader code use alternative data stores, we'd need to replace the fs calls with middleware that doesn't strictly speaking care about what it's interfacing with, and simply knows how to purely read and write json based on a "filename" string key and an env-based option that tells it which service to make use of (filesystem, S3, mongo, whatever)

That should be doable, and if you want to preempt me working on that and try to make that happen: cool! Otherwise, I'm going to have to get to this when I can get to it. The last few weeks have been crunch time at work, and I've not really had time to work on any of my own npm packages and sites =(

Wildhoney commented 9 years ago

I very much appreciate the response, @Pomax :+1: Sadly I don't have the time either to implement an agnostic I/O middleware, but will keep this ticket open for if I do have the time in the near future. Thanks!