add "import -"

[mattn]

This is very experimental feature but ndb search should be possible to used while ndb import.

[jb55]

On Sun, Dec 03, 2023 at 11:33:57PM +0900, Yasuhiro Matsumoto wrote:

--- a/nostrdb.c +++ b/nostrdb.c @@ -3410,6 +3410,72 @@ int _ndb_process_events(struct ndb ndb, const char ldjson, size_t json_len, in return 1; }

+static char read_line(FILE fp) +{

• char *line = NULL;
• size_t maxlen = 0;
• for(;;)
• {
• char sep, block;
• maxlen += BUFSIZ;
• line = realloc(line, maxlen + 1);
• if (line == NULL)
• break;
• block = line + maxlen - BUFSIZ;
• if (fgets(block, BUFSIZ + 1, fp) == NULL)
• {
• if (block == line)
• {
• free(line);
• line = NULL;
• }
• break;
• }
• sep = strchr(block, '\n');
• if (sep != NULL)
• {
• *sep = 0x0;
• if (sep != block)
• {
• if ('\r' == *(--sep))
• *sep = 0x0;
• }
• break;
• }
• }
• return line; +}

• +int ndb_process_events_stream(struct ndb ndb, FILE fp) +{ +#if DEBUG

• int processed = 0; +#endif
• while (!feof(fp)) {
• char *line = read_line(fp);
• if (line == NULL)
• break;
• if (!ndb_process_event(ndb, line, strlen(line))) {
• ndb_debug("ndb_process_event failed\n");
• free(line);
• return 0;
• }
• free(line); +#if DEBUG
• processed++; +#endif
• }

• +#if DEBUG

• ndb_debug("ndb_process_events: processed %d events\n", processed); +#endif
• return 1; +}

Why not just use "getline"?

int ndb_process_events_stream(struct ndb ndb, FILE fp) { char *line = NULL; size_t len = 0; ssize_t nread;

while ((nread = getline(&line, &len, stdin)) != -1) {
    if (line == NULL)
        break;
    ndb_process_event(ndb, line, strlen(line));
}

if (line)
    free(line);

}

[jb55]

On Sun, Dec 03, 2023 at 11:38:41PM +0900, Yasuhiro Matsumoto wrote:

---

nostrdb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/nostrdb.c b/nostrdb.c index b8db79d..b14ffac 100644 --- a/nostrdb.c +++ b/nostrdb.c @@ -3470,7 +3470,7 @@ int ndb_process_events_stream(struct ndb ndb, FILE fp) }

if DEBUG

• ndb_debug("ndb_process_events: processed %d events\n", processed);

• ndb_debug("ndb_process_events_stream: processed %d events\n", processed);

  endif

  return 1;

you can just squash this into the first patch

[mattn]

sure.

[mattn]

BTW, Why ndb accept three element items? ["EVENT", "s", {...}]

[jb55]

On Mon, Dec 04, 2023 at 06:12:39PM -0800, mattn wrote:

BTW, Why ndb accept three element items? ["EVENT", "s", {...}]

originally nostrdb was designed as a client for consuming events from relays, so this is the format relays returned results to be ingested into nostrdb.

These days it's a bit of a client/relay hybrid, so we should probably just have a bare note import, or at least an ingester that supports events of all kinds:

["EVENT", {}] ["EVENT", "subid", {}] {}

It's something I will improve in the future.