search

perry-mitchell / ulidx

ULID generator for NodeJS and the browser
MIT License
281 stars 17 forks source link

Need a method to get ulid as uuid string #8

Closed budarin closed 2 years ago

budarin commented 2 years ago

UUID is more productive data format in db, so we need to have ulid as uuid

Cold you add a method to get ulid as uuid, please?

perry-mitchell commented 2 years ago

Could you explain how UUID is more productive in terms of a DB? ULID is shorter string wise.. I guess you probably mean regarding sharding?

I’m not sure that converting ULID to UUID is within the scope of this library. If you want a UUID, why not use a UUID library?

budarin commented 2 years ago

UUID is stored in Postgres as 16 bytes not as a string with 26 bytes length

budarin commented 2 years ago

https://www.postgresql.org/docs/9.1/datatype-uuid.html

https://dba.stackexchange.com/questions/89429/would-index-lookup-be-noticeably-faster-with-char-vs-varchar-when-all-values-are/89433#89433

perry-mitchell commented 2 years ago

@budarin Would you consider making a PR? I'd happily consider the feature if someone were to handle it. It's not on my roadmap at least.

samal-rasmussen commented 2 years ago

Packages to convert base32 encoded strings to buffers and back already exist:

import { ulid } from 'ulidx';
import { CrockfordBase32 } from 'crockford-base32';

const ulidString = ulid();
console.log('ulid', ulidString);

const buffer = CrockfordBase32.decode(ulidString);
console.log('decoded', new Uint8Array(buffer).toString());

const res = CrockfordBase32.encode(buffer);
console.log('encoded', res);

Result:

ulid 01GA1GBQMCQC2HRD3E2HA9PJRC
decoded 1,130,131,5,222,140,187,5,28,52,110,20,84,155,75,12
encoded 01GA1GBQMCQC2HRD3E2HA9PJRC

https://stackblitz.com/edit/js-bj7btn?file=index.js

stoickeyboard commented 1 year ago

I'm adding more info here because I ran into this as well. If you're using Postgres there are benefits to storing ULIDs as UUIDs; Using 16bytes vs 26bytes is one of them. To expand on @samal-rasmussen 's response you can get a UUID string like so 

import { ulid } from 'ulidx';
import { CrockfordBase32 } from 'crockford-base32';
import * as uuidBuffer from 'uuid-buffer'

const ulidString = ulid();
console.log("ulid", ulidString)

const buffer = CrockfordBase32.decode(ulidString);
console.log("uuid", uuidBuffer.toString(buffer))

The uuidBuffer.toString function looks pretty simple 

function toString(buffer) {
    if (buffer.length != 16)
        throw new Error("Invalid buffer length for uuid: " + buffer.length);
    if (buffer.equals(Buffer.alloc(16)))
        return null; // If buffer is all zeros, return null
    var str = buffer.toString('hex');
    return str.slice(0, 8) + "-" + str.slice(8, 12) + "-" + str.slice(12, 16) + "-" + str.slice(16, 20) + "-" + str.slice(20);
}

source: https://github.com/davidtsai/uuid-buffer

Alternatively you can use Ulid from id128 found here https://github.com/aarondcohen/id128#ulid

And with that library you can get the uuid representation like so 

const { Ulid } = require('id128');

// Returns Buffer
Ulid.generate()

// Returns UUID string
Ulid.generate().toRaw()

// Returns ULID string
Ulid.generate().toCanonical()

That library also has Ulid Monotonic

loretoparisi commented 1 year ago

Packages to convert base32 encoded strings to buffers and back already exist:

import { ulid } from 'ulidx';
import { CrockfordBase32 } from 'crockford-base32';

const ulidString = ulid();
console.log('ulid', ulidString);

const buffer = CrockfordBase32.decode(ulidString);
console.log('decoded', new Uint8Array(buffer).toString());

const res = CrockfordBase32.encode(buffer);
console.log('encoded', res);

Result:

ulid 01GA1GBQMCQC2HRD3E2HA9PJRC
decoded 1,130,131,5,222,140,187,5,28,52,110,20,84,155,75,12
encoded 01GA1GBQMCQC2HRD3E2HA9PJRC

https://stackblitz.com/edit/js-bj7btn?file=index.js

@samal-rasmussen It's worth to note that using a monotonic ulid:

    const ulidString = ulid.monotonic();
    console.log("ulid", ulidString)
    const buffer = CrockfordBase32.decode(ulidString);
    console.log('decoded', new Uint8Array(buffer).toString());

this will result in

ulid 01H8KGF9HXVJVQSPNGAXE9KG82
decoded 0,98,137,193,233,143,119,45,223,54,172,21,215,38,112,64,128
encoded 01H8KGF9HXVJVQSPNGAXE9KG8200
loretoparisi commented 1 year ago

I'm adding more info here because I ran into this as well. If you're using Postgres there are benefits to storing ULIDs as UUIDs; Using 16bytes vs 26bytes is one of them. To expand on @samal-rasmussen 's response you can get a UUID string like so

import { ulid } from 'ulidx';
import { CrockfordBase32 } from 'crockford-base32';
import * as uuidBuffer from 'uuid-buffer'

const ulidString = ulid();
console.log("ulid", ulidString)

const buffer = CrockfordBase32.decode(ulidString);
console.log("uuid", uuidBuffer.toString(buffer))

The uuidBuffer.toString function looks pretty simple

function toString(buffer) {
    if (buffer.length != 16)
        throw new Error("Invalid buffer length for uuid: " + buffer.length);
    if (buffer.equals(Buffer.alloc(16)))
        return null; // If buffer is all zeros, return null
    var str = buffer.toString('hex');
    return str.slice(0, 8) + "-" + str.slice(8, 12) + "-" + str.slice(12, 16) + "-" + str.slice(16, 20) + "-" + str.slice(20);
}

source: https://github.com/davidtsai/uuid-buffer

Alternatively you can use Ulid from id128 found here https://github.com/aarondcohen/id128#ulid

And with that library you can get the uuid representation like so

const { Ulid } = require('id128');

// Returns Buffer
Ulid.generate()

// Returns UUID string
Ulid.generate().toRaw()

// Returns ULID string
Ulid.generate().toCanonical()

That library also has Ulid Monotonic

@stoickeyboard this will fail when using monotonic ulid. In fact while this will work

function uuidBuffer(buffer) {
        if (buffer.length != 16)
            throw new Error("Invalid buffer length for uuid: " + buffer.length);
        if (buffer.equals(Buffer.alloc(16)))
            return null; // If buffer is all zeros, return null
        var str = buffer.toString('hex');
        return str.slice(0, 8) + "-" + str.slice(8, 12) + "-" + str.slice(12, 16) + "-" + str.slice(16, 20) + "-" + str.slice(20);
    }

    const ulidString = ulid.ulid();
    console.log("ulid", ulidString)
    const buffer = CrockfordBase32.decode(ulidString);
    console.log('decoded', new Uint8Array(buffer).toString());
    const res = CrockfordBase32.encode(buffer);
    console.log('encoded', res);
    console.log("uuid", uuidBuffer(buffer))

getting the right result

ulid 01H8KGKBKZJBXA45SQWS9CGXW8
decoded 0,98,137,194,107,159,228,190,168,133,205,249,148,178,29,226
encoded 01H8KGKBKZJBXA45SQWS9CGXW8
uuid 006289c2-6b9f-e4be-a885-cdf994b21de2

switching to monotonic:

const ulidString = ulid.monotonic();

will cause

ulid 01H8KGV9S2YRCX8J6P2TRRT0JY
decoded 0,98,137,195,105,200,189,134,117,18,53,133,172,99,64,151,128
encoded 01H8KGV9S2YRCX8J6P2TRRT0JY00
(node:83614) UnhandledPromiseRejectionWarning: Error: Invalid buffer length for uuid: 17
loretoparisi commented 1 year ago

Apparently it will fail (randomly) with a UnhandledPromiseRejectionWarning: Error: Invalid buffer length for uuid: 17 even when non monitonic ulid has been generated:

ulid 01H8KGZ0TABWG23ZATB80NFCMS
decoded 0,98,137,195,224,210,151,200,8,127,86,150,128,85,236,166,64
encoded 01H8KGZ0TABWG23ZATB80NFCMS00
(node:84065) UnhandledPromiseRejectionWarning: Error: Invalid buffer length for uuid: 17

using

function uuidBuffer(buffer) {
        if (buffer.length != 16)
            throw new Error("Invalid buffer length for uuid: " + buffer.length);
        if (buffer.equals(Buffer.alloc(16)))
            return null; // If buffer is all zeros, return null
        var str = buffer.toString('hex');
        return str.slice(0, 8) + "-" + str.slice(8, 12) + "-" + str.slice(12, 16) + "-" + str.slice(16, 20) + "-" + str.slice(20);
    }

    const ulidString = ulid.ulid(); // randomly fails with Error: Invalid buffer length for uuid: 17
    console.log("ulid", ulidString)
    const buffer = CrockfordBase32.decode(ulidString);
    console.log('decoded', new Uint8Array(buffer).toString());
    const res = CrockfordBase32.encode(buffer);
    console.log('encoded', res);
    console.log("uuid", uuidBuffer(buffer))
samal-rasmussen commented 1 year ago

@loretoparisi in your example you use ulid.monotonic(). Where is this from? It doesn't seem to be in ulidx.

When I use monotonicFactory from ulidx I get a successfull roundtrip:

import { monotonicFactory } from 'ulidx';
import { CrockfordBase32 } from 'crockford-base32';

const ulid = monotonicFactory();
const ulidString = ulid();

console.log('ulid', ulidString);
const buffer = CrockfordBase32.decode(ulidString);
console.log('decoded', new Uint8Array(buffer).toString());
const res = CrockfordBase32.encode(buffer);
console.log('encoded', res);
ulid 01H8KMSMD9PMXQFA71NMMZCF3X
decoded 1,138,39,76,209,169,181,59,119,168,225,173,41,246,60,125
encoded 01H8KMSMD9PMXQFA71NMMZCF3X
loretoparisi commented 1 year ago

@loretoparisi in your example you use ulid.monotonic(). Where is this from? It doesn't seems to be in ulidx.

When I use monotonicFactory from ulidx I get a successfull roundtrip:

import { monotonicFactory } from 'ulidx';
import { CrockfordBase32 } from 'crockford-base32';

const ulid = monotonicFactory();
const ulidString = ulid();

console.log('ulid', ulidString);
const buffer = CrockfordBase32.decode(ulidString);
console.log('decoded', new Uint8Array(buffer).toString());
const res = CrockfordBase32.encode(buffer);
console.log('encoded', res);
ulid 01H8KMSMD9PMXQFA71NMMZCF3X
decoded 1,138,39,76,209,169,181,59,119,168,225,173,41,246,60,125
encoded 01H8KMSMD9PMXQFA71NMMZCF3X

From the original ulid package, it should be the same. Btw I have actually found that, randomly it fails if you retries more often, like in this case:

ulid 01H8KGZ0TABWG23ZATB80NFCMS
decoded 0,98,137,195,224,210,151,200,8,127,86,150,128,85,236,166,64
encoded 01H8KGZ0TABWG23ZATB80NFCMS00
(node:84065) UnhandledPromiseRejectionWarning: Error: Invalid buffer length for uuid: 17

You can see how apparently const buffer = CrockfordBase32.decode(ulidString); is adding 0 padding to the decoded string...

samal-rasmussen commented 1 year ago

The ulid package? That hasn't been updated in 6 years... ulidx exists because that one is abandoned...

But I tried it and I can't get ulid.monotonic from it either: https://stackblitz.com/edit/js-n52gtt?file=index.js

import * as ulid from 'ulid';
import { CrockfordBase32 } from 'crockford-base32';

function uuidBuffer(buffer) {
  if (buffer.length != 16)
      throw new Error("Invalid buffer length for uuid: " + buffer.length);
  if (buffer.equals(Buffer.alloc(16)))
      return null; // If buffer is all zeros, return null
  var str = buffer.toString('hex');
  return str.slice(0, 8) + "-" + str.slice(8, 12) + "-" + str.slice(12, 16) + "-" + str.slice(16, 20) + "-" + str.slice(20);
}

const ulidString = ulid.monotonic();
console.log("ulid", ulidString)
const buffer = CrockfordBase32.decode(ulidString);
console.log('decoded', new Uint8Array(buffer).toString());
const res = CrockfordBase32.encode(buffer);
console.log('encoded', res);
console.log("uuid", uuidBuffer(buffer))

This just fails with ulid.monotonic is not a function. It does have ulid.monotonicFactory and if I use that, then it works just fine. I tried many times.

Maybe just use ulidx?

loretoparisi commented 1 year ago

The ulid package? That hasn't been updated in 6 years... ulidx exists because that one is abandoned...

But I tried it and I can't get ulid.monotonic from it either: https://stackblitz.com/edit/js-n52gtt?file=index.js

import * as ulid from 'ulid';
import { CrockfordBase32 } from 'crockford-base32';

function uuidBuffer(buffer) {
  if (buffer.length != 16)
      throw new Error("Invalid buffer length for uuid: " + buffer.length);
  if (buffer.equals(Buffer.alloc(16)))
      return null; // If buffer is all zeros, return null
  var str = buffer.toString('hex');
  return str.slice(0, 8) + "-" + str.slice(8, 12) + "-" + str.slice(12, 16) + "-" + str.slice(16, 20) + "-" + str.slice(20);
}

const ulidString = ulid.monotonic();
console.log("ulid", ulidString)
const buffer = CrockfordBase32.decode(ulidString);
console.log('decoded', new Uint8Array(buffer).toString());
const res = CrockfordBase32.encode(buffer);
console.log('encoded', res);
console.log("uuid", uuidBuffer(buffer))

This just fails with ulid.monotonic is not a function. It does have ulid.monotonicFactory and if I use that, then it works just fine. I tried many times.

Maybe just use ulidx?

thank you, let me check again!