apache / age

Graph database optimized for fast analysis and real-time data processing. It is provided as an extension to PostgreSQL.
https://age.apache.org
Apache License 2.0
3.1k stars 408 forks source link

Need postgres with graph extensions running on windows OS #1542

Closed MironAtHome closed 5 months ago

MironAtHome commented 9 months ago

I have a graph kit with custom stored procedures, but it's a subpar to true graph engine. Any assistance building this code on Windows OS would be greatly appreciated.

MironAtHome commented 8 months ago

I got the application to compile for branch release/PG15/1.5.0 If it works out will create a fork and some instructions.

jrgemignani commented 8 months ago

@MironAtHome Thank you! This would definitely be helpful to others.

MironAtHome commented 8 months ago

Just finished port for PG14 version 1.5.0 Will be running regression tests and update with quality check. A few warnings regarding cast from 64 bit long long value type to 32 bit long integer type is something will be checking. Please, look over switch to postgres memory / string handling functions from malloc and various to palloc0 family. Fork is there to follow, just in case, here URL with branch https://github.com/MironAtHome/agewin/tree/port/win32msvc/PG14/1.5.0 For now all code is static will add build scripts later, after port for PG16. Need those running somewhat immedaite.

MironAtHome commented 8 months ago

Similar for Postgres 16 https://github.com/MironAtHome/agewin/tree/port/win32msvc/PG16/1.5.0 Any help testing ( within scope of port and some function substitution ) greatly appreciated.

MironAtHome commented 8 months ago

When running regression I am seeing this error: cache lookup failed for type 190232 It would seem related to my missing some component in the extension creation SQL script. Debugger session seems to help with some of the troubleshooting, but it would take from me a bit more to ram up. Any suggestion is greatly appreciated. Error is related to the very first regression script statement: SELECT * FROM ag_catalog.create_graph('ag_graph_1');

While restarting postgres engine seems to fix the issue, it would really hard to do in production. Granted, the cause of the error could be some error I had while putting together extension building SQL script, this seems to be a corner case that would be very nice to handle for someone like myself, just starting to work with Age graph database extension for Postgres. Any insight is very much appreciated. May be my email address would help: mironathome at hotmail.com Feel free to drop a line.

jrgemignani commented 8 months ago

@MironAtHome Unfortunately, I haven't done Windows coding in a long while and can't help. I will see if any others on the team might be able to assist you.

jrgemignani commented 8 months ago

@muhammadshoaib ?

MironAtHome commented 8 months ago

Thank you for checking on my messages. I am mostly looking for just running regression and being par for the course with what it produces on linux. I got the functions running, the handle intercept is not working for now, I think I need to redefine the handle related function set along the lines the macro PGMODULEEXPORT was redefined. Will get back once I try and have results. It would help if you could please share email or some other token to communicate more detailed technical messages, so that my explorations in how the engines works internally was more of an engineering channel.

MironAtHome commented 8 months ago

Just pushed two comments to branches PG14/1.5.0 PG16/1.5.0 Looks like hook functionality is now intact. Again, any help with regression greatly appreciated.

MironAtHome commented 8 months ago

It looks like most of cypher functionality and C functions are working under branch port/win32msvc/PG14/1.5.0 in the fork CSV loading module is in works As as aside, I am not 100% convinced that yet another API CSV within graph module in C is easy to maintain, platform wise. Would you be open to consider loading files into Postgres tables for regression sakes and than ingesting data into graph from table? This might help to obviate some of the sharp(er) corners of dealing with character sets, since it would match what Postgres loads. By itself, I believe, Postgres has both, LOAD feature for the CSV files, that should be able to handle any character set comfortably, and 3rd party extensions handling CSV file(s).

jrgemignani commented 8 months ago

@MironAtHome I will ping someone again on the team who might be able to help.

muhammadshoaib commented 8 months ago

Hi,

Could you please provide the information about your environment weather you are running AGE on windows using Visual Studio compiler or you are using WSL. We have done a testing on WSL long time ago.

Your feedback will help in determining the issue and will lead to eventual solution.

Thanks. MS

MironAtHome commented 8 months ago

Thanks John! Greetings MS, Windows OS, dev station(s) is Microsoft Windows 11 Pro running Postgres release 14 till latest, thinking to synch to Postgres 16. Compiler - Microsoft cl.exe, however, I use Visual Studio 2022 and drive build using Microsoft project ( all checked in under fork's branches ). I drive development under branch port/win32msvc/PG14/1.5.0, here is branches I use port/win32msvc/PG14/1.5.0 port/win32msvc/PG16/1.5.0 after I like what I have in 14 I sync to 16, it's a bit delicate merge due to differences. I have setup ( yesterday ) cross compilation environment running Linux CentOS 8, just applied Postgres 16 compiled from source, stable 16 branch paired with matching Age extension for cross check of behaviors. So, you can see, no WSL is used in either case. I am not 100% comfortable using subsystems for eventual production grade workloads, so, WSL will have to be left away from this exploration.

uhayat commented 8 months ago

I started working on this, hopefully I will be done in couple of days with minimal changes in AGE to make it cross platform.

MironAtHome commented 8 months ago

Hello, I really appreciate effort. My fork just got all the changes needed for the code. Please find is the updated branch: https://github.com/MironAtHome/age/tree/port/win32msvc/PG14/1.5.1 There is still a bit of work left to do

  1. extend changes to Postgres release 16
  2. document steps needed to setup windows flavor of flex, possibly following pattern established by windows build for Postgres database
  3. add build targets for the flex and bison for Visual Studio project file
  4. and wrap up on drawing table based ingestion, regression
  5. and style cleanup Please do let me know if I can be of any assistance, I am hoping to wrap up work on my fork by end of this week. Thank you greatly for your help!
github-actions[bot] commented 5 months ago

This issue is stale because it has been open 45 days with no activity. Remove "Abondoned" label or comment or this will be closed in 7 days.

github-actions[bot] commented 5 months ago

This issue was closed because it has been stalled for further 7 days with no activity.

MironAtHome commented 4 months ago

I have updated build scripts and branches to sufficient quality to get a quality build with certain well known hacks related to Windows SDK. Here is two branches of the fork: https://github.com/MironAtHome/age/tree/port/win32msvc/PG14/1.5.1 https://github.com/MironAtHome/age/tree/port/win32msvc/PG16/1.5.1 Those are branched off of PG14 and PG16 respectively. Here is build instructions ( copied over from issue #1877 Please see here

Please run tools\msvc\build.pl from under root Please note, you may run into some SDK conflicts, please let me know. Need to modify some of the SDK files, here is how it looked on my machine:

C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\um\winternl.h
STRING ->  STRING_TYPE;
typedef STRING_TYPE *PSTRING_TYPE;

#define STRING STRING_TYPE
#define PSTRING PSTRING_TYPE

C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\shared\wtypes.h
DECIMAL -> T_DECIMAL
typedef DECIMAL *LPDECIMAL; -> typedef T_DECIMAL *LPDECIMAL;
#define DECIMAL T_DECIMAL

C:\Program Files (x86)\Windows Kits\10\Include\10.0.22621.0\um\winnt.h
CHAR; -> CHAR_TYPE
typedef char CHAR_TYPE;
#define CHAR CHAR_TYPE

Please note, if you have multiple Windows SDK installed, you may want to specify in the .vcxproj file SDK you would like to target, for example:

<PropertyGroup Label="Globals">
  ...
  <WindowsTargetPlatformVersion>10.0.22621.0</WindowsTargetPlatformVersion>
  ...
</PropertyGroup>