Getting Readline working in PSQL

[chrisant996]

But it is currently crashing because "fd" is -1; (the file descriptor). So, I had to HACK a break point, and skip the IsTTY(fd) check, and just run _getch();

Stack is: rl_read_key() -> rl_get_char() [input.c circa line 514] then it calls c = (*rl_getc_function) (rl_instream);

    which calls rl_getc(FILE *stream)

    And this *stream is not properly initialized.
    I think in MY READLINE_INITIALIZE I need to set this to stdin somehow?

Take a look at the sample repo. I don't know exactly what is causing the problems described above, but I think they're probably self-inflicted wounds. 😉

You mentioned earlier something about needing to declare your own copies of some Readline variables. If that's still being done, then don't. That can only create problems.

Originally posted by @chrisant996 in https://github.com/chrisant996/clink/issues/377#issuecomment-1322926264

[chrisant996]

@kirkw regarding rl_instream:

• rl_instream is initialized to NULL in readline.c.
• readline_initialize_everything() sets it to stdin if it hasn't already been set to some other FILE* stream.
• readline_initialize_everything() is called inside rl_initialize(), which is called inside readline().

So, if you're saying that stdin is broken and has a -1 file descriptor, then that's an issue external to Readline.

Maybe look for anywhere in the host program that may have explicitly set rl_instream to something.

[kirkw]

Okay, rl_instream is set properly. Traced it.

but the call "fileno(stream)" is returning -1. I am not sure if that is invalid or not. some hiccups in that code.

I still have NO source for the "readline" folder in this repo?

[chrisant996]

It's a git submodule. I think you just need to cd readline and git submodule update? Or maybe git checkout readline-8.2? I don't have a lot of experience with submodules, so off the top of my head I don't know the exact step to take.

[kirkw]

Ah... it's from the GIT for readline?

[kirkw]

Okay, git submodule init git submodule update

That worked! (Thanks I learned something else)

[kirkw]

@chrisant996 Okay, it compiled, but fails to link... Are there any other libraries or things you have. I recognize some of these error messages.

OMG, we are so close!

Thank you again. I owe you...

[chrisant996]

@kirkw it looks like you didn't apply the patch file after cloning the repo. See the README.md file for more info.

[kirkw]

Right you were. I read it a while back in your post, and forgot to apply it. I am down to 3 errors... (Which I fixed... so I am able to run the program and see the Green Prompt) My deepest thanks. Sorry for the last couple of screwups!

Strange. Although bind.c calls

define READLINE_LIBRARY

The conditional code in 3 places (keymaps, xmalloc, and readline.h) were HIGHLIGHTING the right code in VS, but actually including the readline/rlstdc.h I tweaked it manually:

if defined (READLINE_LIBRARY) || 1

just an FYI. I don't think they included bind.c

Severity Code Description Project File Line Suppression State Error C1083 Cannot open include file: 'readline/rlstdc.h': No such file or directory readline C:\rl\readline_win32_sample\readline\xmalloc.h 28
Error C1083 Cannot open include file: 'readline/rlstdc.h': No such file or directory compat C:\rl\readline_win32_sample\readline\readline.h 35

Again, it's fixed...

[chrisant996]

@kirkw

Strange. Although bind.c calls

define READLINE_LIBRARY

The conditional code in 3 places (keymaps, xmalloc, and readline.h) were HIGHLIGHTING the right code in VS, but actually including the readline/rlstdc.h I tweaked it manually:

if defined (READLINE_LIBRARY) || 1

just an FYI.

Something (or things) are not right.

If you have to make any changes beyond applying the patch, then something's not right.

I don't think they included bind.c

Can you clarify the statement?

Who?

What does "included" mean in this context?

Update: Do you mean the files keymaps.c and etc? .c files almost never #include other .c files. #define directives are local to the overall file scope they're in. E.g. #include is within the scope of some other file.

I'm unclear on what you're observing, and where. Can you clarify?

Severity Code Description Project File Line Suppression State Error C1083 Cannot open include file: 'readline/rlstdc.h': No such file or directory readline C:\rl\readline_win32_sample\readline\xmalloc.h 28
Error C1083 Cannot open include file: 'readline/rlstdc.h': No such file or directory compat C:\rl\readline_win32_sample\readline\readline.h 35

That means the include directories are declared incorrectly in the project files. Did you make changes to the premake.lua file?

[kirkw]

Sorry, I included the errors with the line numbers...

readline.h line 35: -- The || 1 is what I added, otherwise it was using the other block for some reason.

if defined (READLINE_LIBRARY) || 1
#  include "rlstdc.h"
#  include "rltypedefs.h"
#  include "keymaps.h"
#  include "tilde.h"
#else
#  include <readline/rlstdc.h>
#  include <readline/rltypedefs.h>
#  include <readline/keymaps.h>
#  include <readline/tilde.h>
#endif

[chrisant996]

I see. Don't do that.

The issue is that something is wrong with the list of include directories. But it makes me think the local repo is different than it should be -- I had already addressed this.

What changes have you made in the repo? There should be none, other than applying the supplied patch file, which should be done via the git apply command.

What compiler are you using, and what premake5 command did you run?

[kirkw]

I actually switched folders and did all the git stuff. Then ran Premake5 cleanly.

FWIW. I made no other changes. I was hoping a simple compile would work. You could test it by creating a new folder, and doing the git, git submodule, git update, premake5 and compiling.

NOW, if there is a chance that the READLINE_LIBRARY constant was defined INSIDE your project DEFINES, but somehow PREMAKE did not include them. This could cause it.

Let me try that, to see if I can undo the change!

Again... Really Appreciate the help.

I have this working with TAB and TAB-TAB inside of PSQL based on your example.

Interesting that the READ() function (if you are NOT in binary mode), reads an entire line. Then RE-READS the line, and processes it character by character...

Let me try to fix this so I am working on a CLEAN copy.

[kirkw]

Okay... The COMPAT project needed to DEFINE "READLINE_LIBRARY" Once I did that, I was able to remove the || 1

So, it was working with regards to READLINE compiling. Now the PREPROCESSOR Definitions are:

HAVE_CONFIG_H
HANDLE_MULTIBYTE
DEBUG
_DEBUG
_HAS_EXCEPTIONS=0
_CRT_SECURE_NO_WARNINGS
_CRT_NONSTDC_NO_WARNINGS
READLINE_LIBRARY

[chrisant996]

FWIW. I made no other changes. I was hoping a simple compile would work. You could test it by creating a new folder, and doing the git, git submodule, git update, premake5 and compiling.

That's my plan this morning, to see if I described steps wrongly or something.

NOW, if there is a chance that the READLINE_LIBRARY constant was defined INSIDE your project DEFINES, but somehow PREMAKE did not include them. This could cause it.

Let me try that, to see if I can undo the change!

I'll take a look this morning. But in the premake.lua file I specifically set up the include directories so that including readline/file.h works.

Interesting that the READ() function (if you are NOT in binary mode), reads an entire line. Then RE-READS the line, and processes it character by character...

Binary mode is only about making it so Enter can be read, otherwise the C runtime gets confused and strips the ^M (CR) character thinking it's part of a CRLF sequence, and then it just returns an empty string.

The console mode flag about processed input is what controls how reading console input works; a line at a time or a character at a time.

[chrisant996]

Okay... The COMPAT project needed to DEFINE "READLINE_LIBRARY" Once I did that, I was able to remove the || 1

So, it was working with regards to READLINE compiling. Now the PREPROCESSOR Definitions are:

HAVE_CONFIG_H
HANDLE_MULTIBYTE
DEBUG
_DEBUG
_HAS_EXCEPTIONS=0
_CRT_SECURE_NO_WARNINGS
_CRT_NONSTDC_NO_WARNINGS
READLINE_LIBRARY

No... The following line in the premake.lua file is what makes the compat directory not need the code modifications you've proposed.

https://github.com/chrisant996/readline_win32_sample/blob/b50422eaae25e1e28551a0274374e8ae507b192f/premake5.lua#L88

When you say that the compat directory needs the proposed change ... do you mean in order to build the sample? Because no. If you need it to build the sample, something has been done incorrectly.

Or do you mean in order to build after copying the files into PSQL? Because no, that would mean you don't have the include directories set up correctly in PSQL.

[chrisant996]

@kirkw Here is how to clone the repo, since it contains a submodule. I suspect that the initial cloning is where things went wrong.

:: CLONE THE REPO
git clone --recurse-submodules https://github.com/chrisant996/readline_win32_sample.git
cd readline_win32_sample

:: APPLY THE PATCH
cd readline
git apply ../readline-8.2-win32-modifications.patch
cd ..

:: RUN PREMAKE TO CREATE PROJECTS
premake5 vs2019

:: BUILD
msbuild

[chrisant996]

@kirkw two things:

1. I've pushed an adjustment to clarify the READLINE_LIBRARY stuff.
2. I've pushed an updated patch file which is the patch I'll send to Chet (Readline maintainer) for review.

You should delete the local repo and start over, running the steps from here (i.e. my previous reply) to clone the repo properly and apply the latest patch.

[kirkw]

@chrisant996 Awesome. I will do this in a little bit.

[chrisant996]

I've updated the README.

The last thing I still need to do is provide a custom read callback to handle VT input emulation for all versions of Windows (similar to what Clink does).

[kirkw]

When you say that the compat directory needs the proposed change ... do you mean in order to build the sample? Because no. If you need it to build the sample, something has been done incorrectly.

Or do you mean in order to build after copying the files into PSQL? Because no, that would mean you don't have the include directories set up correctly in PSQL.

Okay. When I right click in VS2022 to build COMPAT project. It would throw an error. When I edited the DEFINES and included that one on the COMPILE line. Then it worked.

This was building your sample program. The issue, AFAICT was that READLINE_LIBRARY was not defined WHILE the COMPAT project was being built. Sorry if that was not clear.

I work in the IDE, so I open the solution that is created, and build from there. (Right click, Build).

But let me start again from a FRESH copy. Now that I have a clue where things are... How they come together.

You've been an incredible asset on this journey... I cannot thank you enough. And I think having a testable readline function out there is awesome. I actually want to expand it to do tab completion as an example at some point...

PS: read() was reading WRONG because it was NOT initialized [their mainloop toggles sigint_interrupt_enabled on every pass] After the first TAB to complete, then it was not working again.

So, I added config_console() to their loop, and that TAB and TAB TAB are working better. Something is still off, but not crashing...

Great Progress

[chrisant996]

Okay. When I right click in VS2022 to build COMPAT project. It would throw an error. When I edited the DEFINES and included that one on the COMPILE line. Then it worked.

This was building your sample program. The issue, AFAICT was that READLINE_LIBRARY was not defined WHILE the COMPAT project was being built. Sorry if that was not clear.

But the premade.lua explicitly already solved that. If you still encountered it, then somehow the repo was not cloned/setup correctly, or you hit some problem in Premake.

But let me start again from a FRESH copy. Now that I have a clue where things are... How they come together.

Sounds good.

I think having a testable readline function out there is awesome.

Agreed. It also potentially gives Chet or others an easy way to verify basic Windows compatibility when making changes in Readline.

I actually want to expand it to do tab completion as an example at some point...

You mean an alternative completion callback, right? Tab completion works as-is.

PS: read() was reading WRONG because it was NOT initialized [their mainloop toggles sigint_interrupt_enabled on every pass] After the first TAB to complete, then it was not working again.

So, I added config_console() to their loop, and that TAB and TAB TAB are working better.

Oh, now you're talking about PSQL, right? It would help to be explicit about context, so I can follow along.

Yes, if you hadn't included anything to alter the console mode, then input wouldn't work. But be careful -- that will break any other input in the program...! Be careful to use BOTH config_console() and unconfig_console() at appropriate points, or things will go wrong.

Something is still off, but not crashing...

If you can describe what's going wrong (even if the description sounds strange), I may be able to recognize what's happening.

Also, the whole UTF8 thing could create problems in PSQL if codepages and encoding conversations aren't handled correctly. You can't just set a program to use UTF8 and be done -- it's easy and likely to have ripple effects and cause breakages downstream.

I added a brief section in the README about the UTF8 considerations.

Great Progress

Indeed!

[kirkw]

Also, the whole UTF8 thing could create problems in PSQL if codepages and encoding conversations aren't handled correctly. You can't just set a program to use UTF8 and be done -- it's easy and likely to have ripple effects and cause breakages downstream.

I added a brief section in the README about the UTF8 considerations.

Yeah, this is a FUTURE issue I will have to deal with. PSQL allows for paying attention to codepages. It warns you if you are in the wrong codepage for the DB settings. (1252 vs 437)

Yes TAB completion (non-augmented) is working in your sample. I was talking about the custom completion

config_console() and unconfig_console()

I am assuming I only need to call unconfig_console() when the program terminates and hands things back to the shell... (Then number of times I call config_console() should not matter)

[chrisant996]

I am assuming I only need to call unconfig_console() when the program terminates and hands things back to the shell... (Then number of times I call config_console() should not matter)

You need to call unconfig_console() whenever there is a possibility for something other than Readline itself to read from stdin.

I don't know enough about when "PSQL terminates" to know whether it's ok to not call unconfig_console() until program termination, but that sounds wrong.

I would expect this kind of flow:

• other PSQL code
• loop
  • other PSQL code
  • config_console()
  • call Readline to get an input line
  • unconfig_console()
  • other PSQL code
• end loop
• other PSQL code

The console mode matters. If you look at the Clink code, it is very careful about saving/overriding/restoring the console modes around every new input line session. Since Clink uses the READLINE_CALLBACKS mode, it is harder to see where a Readline "call" begins and ends, but here it is:

https://github.com/chrisant996/clink/blob/e8df547a7d64e1470e7ca613804baa984e0e0f70/clink/app/src/host/host_cmd.cpp#L646-L654

[chrisant996]

Also, the whole UTF8 thing could create problems in PSQL if codepages and encoding conversations aren't handled correctly. You can't just set a program to use UTF8 and be done -- it's easy and likely to have ripple effects and cause breakages downstream. I added a brief section in the README about the UTF8 considerations.

Yeah, this is a FUTURE issue I will have to deal with. PSQL allows for paying attention to codepages. It warns you if you are in the wrong codepage for the DB settings. (1252 vs 437)

The whole PSQL code base is making an assumption about the current C runtime locale matching the currently active code page.

Calling setlocale(... ".utf8") invalidates that assumption. What you refer to above (warning if in the wrong codepage) is almost certainly assuming that the C runtime is configured to use ACP -- I would bet money that it only verifies the OS is using a matching codepage for the DB (by using GetACP()), and that it assumes the C runtime default locale has not been overridden and changed to something other than ACP. By overriding the C runtime locale, it invalidates the entire rest of the code base's expectations about what encoding conversions are needed, and is almost certainly not prepared for conversions to happen silently and automatically in C runtime functions (non-UTF8 strings passed to C runtime functions will be interpreted as though they are UTF8 even though they're not; and vice versa).

Since UTF8 single byte codepoints mostly match codepage 1252, you're unlikely to notice the problems when using English, French, Italian, German, or Spanish. You'll need to use languages like Japanese, Chinese, Arabic, Korean, Russian, etc for the breakages to be recognizable.

[kirkw]

Since UTF8 single byte codepoints mostly match codepage 1252, you're unlikely to notice the problems when using English, French, Italian, German, or Spanish. You'll need to use languages like Japanese, Chinese, Arabic, Korean, Russian, etc for the breakages to be recognizable.

Perfect... I have a couple of Russians on my team, they can test this. We only use ENGLISH for our work. While that was the PRIMARY GOAL here... The Stretch goal is still (always) to give back to the community.

FYI: After finding that PSQL bug (AutoComplete parsing wrong. It was a "nonconforming preprocessor" not counting VA_ARGS correctly)... I have the CORE of this working in FRESH CODE. MINIMAL set of changes.

Now I will go back and re-grab your updated readline and build that up... Following the README/Notes... A Huge Sigh of Relief. (There is one bug left that is just strange but its PURELY inside PSQL code. Where they copy the buffer to parse it, then free that extra memory. That blows up... But this can be later. I have not even eaten lunch, and it is past dinner. LOL)

[chrisant996]

FYI: After finding that PSQL bug (AutoComplete parsing wrong. It was a "nonconforming preprocessor" not counting VA_ARGS correctly)...

Aha, nice work.

I have the CORE of this working in FRESH CODE. MINIMAL set of changes.

Excellent!

There is one bug left that is just strange but its PURELY inside PSQL code. Where they copy the buffer to parse it, then free that extra memory. That blows up...

I have the postgresql repo. If you want, you can share a branch name, HEAD commit you are in sync with, filename, line number for where the buffer gets copied, and I can take a look as a second pair of eyes.

I have not even eaten lunch, and it is past dinner. LOL

Haha, I am well acquainted with that groove. 😎

[kirkw]

@chrisant996 You have done too much already...

The error I am getting is ONLY in DEBUG mode. It is an assertion on the FREE that the block type is well known. I am calling PURE calls to malloc()/free() that are not wrapped. So I don't know how it expects the header to have some flag set [Usually this is done by allocating an extra byte, and setting the first bytes value, then returning the pointer to the second byte, then when you free, you can backoff one byte and make sure you were never corrupted. But I dont see this code happening anywhere. I've asked one of the PG Guru's I know... I can sleep with it working this well, and just COMMENT out the free in debug. LOL!]

It's actually WORKING. The ONLY difference I see... Is when the psql_completion() is called on a word it does not know. In Linux TAB is ignored. Two TABS spits out a blank line. In Windows TAB in that position is turned into a space. So is the next one... [Your code does not do this]; MINOR stuff at this point...

Here is a screenshot with over 100 tables to choose from...

[kirkw]

@chrisant996 okay, running through the process.

I get this warning from premake:

[chrisant996]

@kirkw, Oh there must be a new recent version of Premake which I don't have yet, which has deprecated the configuration keyword. That warning should be ignorable.

[chrisant996]

The error I am getting is ONLY in DEBUG mode. It is an assertion on the FREE that the block type is well known. I am calling PURE calls to malloc()/free() that are not wrapped. So I don't know how it expects the header to have some flag set [Usually this is done by allocating an extra byte, and setting the first bytes value, then returning the pointer to the second byte, then when you free, you can backoff one byte and make sure you were never corrupted. But I dont see this code happening anywhere. I've asked one of the PG Guru's I know... I can sleep with it working this well, and just COMMENT out the free in debug. LOL!]

Yes, there are several techniques for wrapping an existing heap to add heap validation mechanisms.

But if you're using pure free then how is heap wrapping code executing to fire the assertion? Maybe the free call isn't pure, but the malloc call is accidentally pure when it shouldn't be?

Heap mistakes can be exploitable security vulnerabilities. I'd want to track down what is happening. What if the assertion is pointing out a real mistake? Such as a double free -- which could appear to a wrapped free as though it is merely not the right type, but in fact the heap is being poisoned and memory corruption will begin to occur.

Sharing a file and line number would help.

It's actually WORKING. The ONLY difference I see... Is when the psql_completion() is called on a word it does not know. In Linux TAB is ignored. Two TABS spits out a blank line. In Windows TAB in that position is turned into a space. So is the next one... [Your code does not do this]; MINOR stuff at this point...

Interesting. I'll look up psql_completion in the repo...

[kirkw]

@chrisant996 Okay, so I am still getting a COMPILE error. But I KNOW the cause. One File: tilde.c

It includes "xmalloc.h", but it DOES NOT: #define READLINE_LIBRARY

I had to add it to the DEFINES for the file. Once I do, I can compile...

[kirkw]

Interesting. I'll look up psql_completion in the repo...

If you do. It's psql\tab-complete.c

But it is a 2,000+ Line Long SINGLE if/else statement. I had to break the line to get VS2022 to compile it without an internal stack overflow.

Thanks again... I need a break...

[kirkw]

Heap mistakes can be exploitable security vulnerabilities. I'd want to track down what is happening. What if the assertion is pointing out a real mistake? Such as a double free -- which could appear to a wrapped free as though it is merely not the right type, but in fact the heap is being poisoned and memory corruption will begin to occur.

Yeah, this is my concern here. But this code has NEVER run on windows in 17yrs! So it could be a small compatibility problem, like the ARG counter, and the HUGE IF/ELSE statement. And there is a strange structure, calling a lexer, and when I read the code, I get the feeling something could go wrong... (&var), then var->buf = malloc(); Then var->buf is returned, and freed up in the future. it seems SKETCHY to me. That's why I asked for help from one of the PG devs...

[chrisant996]

@chrisant996 Okay, so I am still getting a COMPILE error. But I KNOW the cause. One File: tilde.c

It includes "xmalloc.h", but it DOES NOT: #define READLINE_LIBRARY

I had to add it to the DEFINES for the file. Once I do, I can compile...

The premake.lua file explicitly compensates for that. There's even a comment about it in the file.

I'm using Premake 5.0.0-alpha15.

I'll download Premake 5.0.0-beta2 and see if I can replicate what you're experiencing.

[kirkw]

I'll download Premake 5.0.0-beta2 and see if I can replicate what you're experiencing.

Yep, I just verified. That's the version I am running. 5.0.0-beta2

I see the line... But here is the INCLUDE setting: (An EXTRA ..?) ....\compat;....\readline;......;%(AdditionalIncludeDirectories)

[kirkw]

@chrisant996 I got it to work in the IDE without the DEFINE: ..\..\compat;..\..\readline;..\..\..;%(AdditionalIncludeDirectories);..\..\

maybe it required 2? '....\' To be added ?

[kirkw]

@chrisant996 Sorry for the spam... This works: includedirs(";")

in the lua script. It adds: "....\;;"

FWIW, your command ".." was adding: "......;" Which we see as the last item...

Thank you, I just learned something about premake!

[chrisant996]

I'll download Premake 5.0.0-beta2 and see if I can replicate what you're experiencing.

Yep, I just verified. That's the version I am running. 5.0.0-beta2

I see the line... But here is the INCLUDE setting: (An EXTRA ..?) ....\compat;....\readline;......;%(AdditionalIncludeDirectories)

I'm not sure why the \ characters got stripped when you copy/pasted.

I'm using Premake 5.0.0-beta2 now, and it works fine for me. And using VS2019 works fine for me. I don't know why it's not working for you; maybe it's some difference between VS2019 and VS2022, though that doesn't sound right.

Anyway, when I change the ".." to "." it still works for me, and based on what you shared it will work for you as well.

So I've pushed that change right now. Please try premake5 vs2022 and force a full rebuild?

[chrisant996]

(&var), then var->buf = malloc(); Then var->buf is returned, and freed up in the future. it seems SKETCHY to me.

@kirkw Please point me at the code... For real, please do. 🙃

Are you using the https://git.postgresql.org/git/postgresql.git repo, or a mirror? What branch are you looking in?

There is no var->buf = malloc anywhere in the master branch in the repo. Do you mean var->buf = digitbuf_alloc(ndigits + 1); in one of the two numeric.c files?

Also, please share the file and line number for the malloc and the free for the assertion you mentioned about well known type for the heap block. Trying to do fuzzy keyword searches on the terms you shared is not finding anything in the repo.

Can you please share file and line number when citing stuff, so that I can follow along? 🙂

[kirkw]

So I've pushed that change right now. Please try premake5 vs2022 and force a full rebuil

This worked. Both Release and Debug. Thank you for addressing.

I have one other issue... I want to INCLUDE "readline_glue.h" But I am getting DUPLICATE definition of "struct timeval"... So I can't. I copied in the prototypes to the functions I call from there... Maybe an IFDEF wrapper?

Thanks...

[kirkw]

Are you using the https://git.postgresql.org/git/postgresql.git repo, or a mirror? What branch are you looking in?

@chrisant996 I am so sorry, I need to get away from the computer now. We are using version 14.6 source code. (My buddy grabbed it from git, not me. He has it zipped up to restore)

The code of interest is in the plsql folder. the files involved are tab-complete.c command.c (line 202 ish): backslashResult HandleSlashCmds(PsqlScanState scan_state, ConditionalStack cstack, PQExpBuffer query_buf, PQExpBuffer previous_buf) {

this is the function that the FREE dies on..

BUT: psqlscanslash.c: around 3,124 psql_scan_slash_command() -- Hits some lexer code???

pgexpbuffer.c: initPQExpBuffer(&mybuf); -- Called from above

Thats the rough call stack.

But it's a lot to look at!

[chrisant996]

So I've pushed that change right now. Please try premake5 vs2022 and force a full rebuil

This worked. Both Release and Debug. Thank you for addressing.

I have one other issue... I want to INCLUDE "readline_glue.h" But I am getting DUPLICATE definition of "struct timeval"... So I can't. I copied in the prototypes to the functions I call from there... Maybe an IFDEF wrapper?

Thanks...

Oh, let me explain:

You only want to include readline_glue.h because you want declarations for config_console() and unconfig_console().

I shared the repo once the standalone sample was working.

I haven't finished it to the point where things are modular enough to be integrated into something other than the sample. For example:

• The "glue" files are private glue for Readline itself; nothing else should use or refer to them.
• The config/unconfig functions need to be moved to a separate header file.
• A custom read callback function needs to be provided so that keyboard input can work on Win 8.1 and lower, and on Win 10 build 16298 and lower.

[kirkw]

Okay, and there is ONE bug in the demo. I pasted in more than 80 characters, and it wraps on the same line, overwiting the prompt. The output line is CORRECT... I assume this can handle a wrapped line? (I never asked, LOL)

[chrisant996]

@kirkw That was fixed already ... it was a bug in the c_sample.c file. Refer to the commit history.

I don't understand how you keep running into so many issues that I already fixed ... I suspect the repo is still not cloned correctly, or something like that.

[chrisant996]

@kirkw re: the config functions, for now I have tweaked the readline_glue.h file so that you can include it elsewhere.

The timeval problem is due to something including winsock.h somewhere (maybe because it was needed, or maybe due to not using #define WIN32_LEAN_AND_MEAN (which by the way speeds up compile time). I've included a workaround for that.

[chrisant996]

I need to get away from the computer now.

I have no expectation about timeframe of replies. 🙂

We are using version 14.6 source code. (My buddy grabbed it from git, not me. He has it zipped up to restore)

The code of interest is in the plsql folder. the files involved are tab-complete.c command.c (line 202 ish): backslashResult HandleSlashCmds(PsqlScanState scan_state, ConditionalStack cstack, PQExpBuffer query_buf, PQExpBuffer previous_buf) {

I'm pretty sure that plsql is a typo and that it's the psql folder, right?

this is the function that the FREE dies on..

I'm guessing you mean this free call near the end of HandleSlashCmds:

    /* if there is a trailing \\, swallow it */
    psql_scan_slash_command_end(scan_state);

    free(cmd);

    /* some commands write to queryFout, so make sure output is sent */
    fflush(pset.queryFout);

The only way I can see (so far) for cmd to not be a heap pointer is for the malloc to fail, which will set the data member to a non-heap pointer, but subsequent code doesn't know it's not a heap pointer. If that case were to occur, it could end up corrupting the heap or causing other malfunctions. I don't see why the allocation would fail, but it certainly looks like the outer calling code will malfunction and corrupt the heap after an allocation failure there.

So, just in case: What is the value of the cmd pointer being freed? What is the value of the oom_buffer_ptr variable?

void
initPQExpBuffer(PQExpBuffer str)
{
    str->data = (char *) malloc(INITIAL_EXPBUFFER_SIZE);
    if (str->data == NULL)
    {
        str->data = unconstify(char *, oom_buffer_ptr); /* see comment above */

Or there could also be a buffer underrun or overrun somewhere, especially in an edge case.

BUT, I'm still trying to find the assert that you said gets hit. I would need to see the assert to understand what state is occurring, and have an idea what code patterns to look for.

I've done all the investigation I can with the info I have so far.

If you're interested, we can try stepping through the debugger together in the cloud machine. We can likely use a memory breakpoint or even Time Travel Debugging to track down what's happening.

[chrisant996]

I pushed an update with a custom input callback to provide VT terminal input emulation on all versions of Windows.

[kirkw]

Or there could also be a buffer underrun or overrun somewhere, especially in an edge case. Here is the assert(). It is a header validation of sorts. I think it is INTERNAL to the MSFT library. So a buffer UNDERRUN would be the case.

The code was: psql.exe

[chrisant996]

Or there could also be a buffer underrun or overrun somewhere, especially in an edge case. Here is the assert(). It is a header validation of sorts. I think it is INTERNAL to the MSFT library. So a buffer UNDERRUN would be the case.

The code was: psql.exe

@kirkw That means the heap has been corrupted -- that's serious; it's not ignorable.

The heap will be corrupted in release builds as well. But since release builds don't fire asserts, the only indication would be that eventually things malfunction in unexpected ways and/or the process crashes or hangs.

The cause might be an underrun, but it could also be other things -- a double-free, or freeing an address that wasn't allocated in the first place, or etc.

I'll be available around 12:30 PM Pacific Time today.

However, I can't use TeamViewer because my employer blocks it, even on my personal laptop (which is registered and managed by the company, so that I can access my email from it). Sidebar: this is the first and only time in my career that a piece software I've wanted to use was actually blocked by my employer. I imagine there's a good reason, but I don't know the details.

Perhaps instead we could use Zoom with screen sharing?

[kirkw]

Perhaps instead we could use Zoom with screen sharing?

Sure, I have a GoToMeeting License if you want to avoid the "company" Zoom, or the 45 minute limitation... Email me... I will send you the link. Or I can create a temporary link, and pop it in here. I am on the East Coast, so I think 12:30pm your time is 3:30pm my time. About 1hr 15 minutes from now.