Add support to fetch content via http and cache locally

[nkhorman]

I've add rudimentary support for fetching remote content via libcurl. Thought / wondered if anyone there might be interested... I'd be happy to see it merged. The code is mildly tested. I'm sure there is plenty of room for improvement / feedback.

[metdos]

Hey @nkhorman,

This project is retired and currently not recommended as a base for new consumers. If you're looking to query semi-structured data, please check the new JSON and JSONB data types built into PostgreSQL.

[nkhorman]

@metdos , Thanks for clarifying the status of the project.

Though I have a question, how does the JSON / JSONB type dynamically retrieve and update foreign content ? Maybe I'm missing something.

Isn't that the whole point of an FDW ?

While this FDW, fetched content from local disk, which is a start, I added the ability to fetch the content from a remote server which is the subject for this pull request.

I was waiting for this request to be merged to then follow up with a pull request for the additional ability to do insert and update operations, which I have finished.

It seems to me, that anyone actually interested in foreign dynamic content, actually wants this functionality.

Oh well, I guess, that's what forks are for.

[metdos]

Hi @nkhorman,

You are right; JSON / JSONB type doesn't allow to operate on foreign content. On the other hand, with the introduction of JSONB in PostgreSQL, we started to see more of our users load their data into JSONB columns and bypass foreign tables. Therefore, we decided to retire this project.

I see that you were actively developing on your fork. Are you planning to keep working on this project? If you plan to keep developing this project, we can add a small note to our readme and guide people to your fork. What do you think?

If you have any questions or feedback for us, could you let us know?

[nkhorman]

hi @metdos ,

I plan to continue to develop this as bugs and needs for features arise, wether that be driven by me or pull requests from others, as time permits.

A readme note pointing or mentioning my fork would be fine.

I'll rename my project as json_fdw2 to demark / denote the transition of your now retired original project, and avoid confusion between the two code bases.

[metdos]

Hi @nkhorman,

Great to hear that you plan to continue developing json_fdw.

I think you can keep the name as json_fdw. You can check our another retired project which is now maintained by EnterpriseDB. https://github.com/citusdata/mongo_fdw https://github.com/EnterpriseDB/mongo_fdw

Let us know your decision about naming, so we can add a link to our readme.

Best, Metin

[nkhorman]

Hi @metdos,

If your using the mongo_fdw "Please note this repository has moved to ..." (https://github.com/nkhorman/json_fdw), instead of the "retired" notation, as an example of what you're going to do in the readme, then I'll keep the original name.

Otherwise, I want it to be clear, that my fork is not retired, hence the fdw2 moniker, to instead denote progression.

Regards

[nkhorman]

Hi @metdos, Ping.

[metdos]

Hi @nkhorman,

We've talked about this inside our team. We think it would be better to use json_fdw2 to differentiate between projects. Sorry for the confusion.

You can let me know when you rename the repository. Then, I can add a link to our readme.

[ozgune]

@nkhorman -- also thank you for using json_fdw and carrying the flag forward with this project!

[nkhorman]

Hi @metdos, I apologize for the delay. I've renamed the project, and updated the README.md. Take a look at it, and let me know of any changes that you'd like to see.

Hi @ozgune , Thanks to Citus Data for building and publishing this! It was a great starting point for what I wanted to do. I hope someone finds what I've done to it to be useful.

[metdos]

Hi @nkhorman,

We removed the retired note and added a moved note to your fork.

Best, Metin

[nkhorman]

Hi @metdos Thanks.