search

cyga / www_fdw

fdw extension for postgres
http://wiki.postgresql.org/wiki/WWW_FDW
123 stars 21 forks source link

Has anybody succeeded ever to port www_fdw to PG13 or later? #46

Open marmor opened 2 years ago

marmor commented 2 years ago

I've been busy with it for several days, and before continuing, I must know if somebody else already passed this journey, to save my time:

In the beginning it didn't compile. After fixing the compilation problems (which was easy), it crashed the server. I had to fix 2 different bugs that each of them crashed the server (less easy). Then I had to find a working example (all of the examples from the EXAMPLES page are obsolete, so I fixed 2 of them - for example the wiki one, where all I had to do is to replace the HTTP by HTTPS). Now it compiles OK, runs, and I track the entire process, with zillion of logs and prints, and 90% of the way looks OK, but the final result is still bogus (e.g. SELECT returns 0 rows, even when the logs from the fdwroutine functions show more rows).

There is no documentation for porting FDW from old PG versions to the recent ones, so it's very difficult for me to continue, but I believe that www_fdw could make a real revolution IF if worked; If many of the APIs in the world become automatically PG tables, it will be a real revolution, so I don't want to give up.

So anybody who passed this journey, but succeeded to run it under 13 or 14, please share your experience, tips, and maybe even code.

Thanks!

jmls commented 1 year ago

did you ever get to fix this ?

marmor commented 7 months ago

did you ever get to fix this ?

Sorry, I just saw your question now...

FDW, if it worked, could be the ONE THING that makes PG the leading DB; Not only that it allows you to port your tables from any DB incrementally thanks to the fact that it mimics all of the other DBs (you don't have to migrate all of the tables at once), but it mimics the entire world, and everything becomes a PG table, from your OS processes to your Docker images and instances to ziliion of other things (like many public APIs, thanks to www_fdw. If it worked...).

So you can understand how disappointed I am...

I can tell you a lot also about my disappointment of the FDW frameworks.

But I'm here to answer your question about what CAN be done, not what can't be done, so let's start:

  1. One option is to adopt Supabase's FDW framework (called "Wrappers"). It has one huge drawback: It's written in Rust. I'm not going to enter a flame war about "what language is better", the problem is that most of the systems are not written in Rust, most of the programmers don't know this language, most of the libraries and modules were not ported to Rust, and in short - it's not standard and is just one more language that you must learn and insert into your project. However, on the other hand, there is also a huge advantage: Supabase is a strong company, developing its product to last and to win, and like us, loved the idea of FDW and suffered the fact that FDW doesn't really work. So you can bet on its framework. So if Rust is not a problem for you, take Wrappers and easily develop anything you want, it will be a piece of cake!

  2. The future is PLJS. Contrary to Rust, Javascript is the most popular language today, and almost anything (except for specific areas like AI) is written or bound to JS. Everybody knows JS, and it's already included in any system. In the beginning, there was PLV8, which is "JS for PG". Once you have a language implementation for PG, the next step is to make an untrusted version of it, and then to make it applicable for writing FDWs. However, the result was terrible: it was so huge, that its size was several times more than the entire PG! I remember that it took me several hours just to build it... So the developers of PLV8 started a new project, called PLJS, that is still in Alpha status, but looks amazing and promising. (And its size is about 1/10 of PLV8...)

  3. So someties in the future, most of us will write FDWs in JS, but what now? So just to have something working, I took Multicorn, and I write my FDWs in Python3. It's not simple, because as almost anything else in the FDW field, Multicorn is almost unsupported, so you have to take a fork of it (Multicorn2) (which is not much better...), and it has many other drawbacks, but I look at it as a temporary workaround till PLJS is ready.

Once PLJS is ready, and support FDW, my dream is to develop a generic FDW, like www_fdw could be, and to publish it.

Till then, if you want a specific API to become a PG table, just write your own FDW for it with Supabase' Wrappers or with Multicorn2, it's much easier than writing FDWs in C (maybe the best idea is to write FDW framework like Multicorn and Wrappers (and the future PLJS...), but in C...).