search

saurabhnanda / odd-jobs

Haskell job queue with admin UI and loads of other features.
https://www.haskelltutorials.com/odd-jobs/
BSD 3-Clause "New" or "Revised" License
75 stars 29 forks source link

JobId should be Int64 #39

Open TomMD opened 4 years ago

TomMD commented 4 years ago

JobIds are currently Int which is potentially limited to 2^29. They should be Int64 at least.

saurabhnanda commented 4 years ago

If my calculation is right, even at one job / sec, it would take 17+ years to reach this limit.

Are you planning to use this instead of a message queue, by any chance?

TomMD commented 4 years ago

Are you planning to use this instead of a message queue, by any chance?

Ding ding!

saurabhnanda commented 4 years ago

Seriously? Are you planning to use odd-jobs instead of a message-queue? How many messages / sec are you expecting?

TomMD commented 4 years ago

Yes, seriously. I'd rather have my jobs persist and not be an in-memory store that could be lost in the event of an outage. I see perhaps 20 message per user per day. If I am super successful and see 10M users then thats 700 message / second spread over the 8 hour work day uniformly.

jkachmar commented 4 years ago

+1 for making JobId an Int64.

Is Int being used to address a particular problem with performance or overhead, or just because it's not an expected use-case for the queue to grow beyond 2^29 entries?

saurabhnanda commented 4 years ago

Is Int being used to address a particular problem with performance or overhead, or just because it's not an expected use-case for the queue to grow beyond 2^29 entries?

I didn't think through this deeply. What is the natural choice for mapping a Postgres integer to a Haskell data-type? How many bits does a Postgres integer use?

TomMD commented 4 years ago

How many bits does a Postgres integer use?

Postgres integer is 32 bits: https://www.postgresql.org/docs/10/datatype-numeric.html

bigint is 64 bits

saurabhnanda commented 4 years ago

So, ideally, if we make this change it should be int <=> Int and bigint <=> Int64, right?

Is there value in making this configurable? Or will it end-up being unnecessary complexity for library users for little gain?

TomMD commented 4 years ago

That makes sense to me, yes. There could be an argument for type JobId = Int64 too.

mrputty commented 4 years ago

FWIW, I don't think there's any value in making this configurable. The representation of the job id is an internal library concern, at least the way the library is structured today. I think just using Int64 seems the most reasonable to me.