Open nelsonic opened 5 years ago
Maybe use something like like a hash, or nanoId?
Or configure the database to generate a non-sequential unique id that is not a uuid when creating the resource?
This is also an interesting blog with some insight into making the uuid
more "pretty" for the user.
This might be overkill but you could also, perhaps (I have not tested this), create a base64 decoding/encoding method that uses a salt or an iv to sign and create a AES192 cipher/decipher of your id
.
Another solution would be to ditch the uuid completely and create the resource id
based on pre-determined criteria that is designed to be user friendly OR have a linked table that would map directly to your uuid
?
@hyprstack great points! Thanks for sharing links and the Coding Horror Blog post; I found the comments especially insightful.
NanoID https://github.com/ai/nanoid (initial commit 5 Aug 2017)
Under the hood it's using crypto.randomFillSync(buffer)
see: https://github.com/ai/nanoid/random.js#L11
Similar to https://github.com/nelsonic/perma (initial commit 14 Feb 2015) which uses
crypto.createHash('sha512')
perma
removes "confusing characters" by default and avoids difficult special chars e.g:~
(tilde)
Nano ID has a good illustration of the Birthday Problem (Probability of ID Collision) to explain ID length:
https://zelark.github.io/nano-id-cc/
Anyone following along needing a recap or intro to the Birthday Paradox, see: https://betterexplained.com/articles/understanding-the-birthday-paradox
If we could configure/program PostgreSQL to auto-generate the non-sequential ID it would be good because the responsibility for managing/preventing collisions would be handled by the DB and we wouldn't need to think about it. 👍
Hashids are good for obfuscating integers to strings and the reversibility is useful for many situations.
The official usecase is for "when you don't want to expose your database ids to the user":
My reasoning for not wanting to use Hashids is twofold:
We could remedy this by providing the charset as the 3rd argument.
hashids.encode
function. This by definition reduces the entropy to the upper bound of JS numbers: 9007199254740991
16 digits.
So if we used Hashids in an app creating 1k records/sec (e.g. in a reasonably popular analytics/chat app) there would be a collision after only 4hours.
I agree that using a cryptographic hash function for creating IDs is a good idea for creating random strings with low chance of collisions.
We considered the use of a salt/iv when hashing - and it's a good suggestion in general when hashing to avoid rainbow table attacks - but it would only be useful in the case where we wanted to make ID reversibility (by an attacker) more difficult. I'm not discounting the use-case of a salt when hashing content for ID creation yet, just not having it for "MVP" because I don't currently envisage ID cracking to be the weakest link in the system. 🤔
I can summarise the requirements I have in mind in the following 3 statements:
I think I need to start writing some code to illustrate my thoughts on this. 💭 |>📝 Thank you very much for chiming in with relevant links and insight @hyprstack 🥇 Hope your weekend is going well. 🎯
Discovered https://github.com/nocursor/ex-cid/ 🎉 while reading: https://github.com/ipld/cid#implementations via: https://github.com/nelsonic/learn-ipfs/issues/6#issuecomment-444067582 Tried to clone and run it locally ... failed. 😞 Opened issue: https://github.com/nocursor/ex-cid/issues/1 (to ask for help from the author!) ⏳
Sadly, after a week of waiting for the author of ex-cid
to respond to this issue: https://github.com/nocursor/ex-cid/issues/1
I feel that I have no other choice ... I cannot afford to keep waiting indefinitely for this to work. ⏳
I can either fork it and dramatically improve the quality of the tests, code and documentation and the wait for the author to respond to the issue, or I can finish my own implementation. 🤔
Sadly, ex-cid
was accepted as the "recommended" elixir implementation in this PR: https://github.com/multiformats/multibase/pull/39 so now in addition to duplicating effort I have to inform the Multiformats community that the rush to publish ex-cid
and subsequent non-responsiveness from the author is hindering adoption ... 😢
I hate it when people rush code and don't make the effort to add reliability guarantees via continuous integration with 100% test coverage and comprehensive examples. I've had to "fill in the gaps" a couple of times before ...
e.g:
But I definitely don't "like" doing it! I would much rather someone else write and maintain this! I want to build Incredible User Experiences/Interfaces not infrastructure...! 🙄
Anyway, GOTO: https://github.com/dwyl/cid/issues/11
More than a month has passed since I opened the issue https://github.com/nocursor/ex-cid/issues/1 informing the author/creator that the code does not compile https://github.com/nocursor/ex-cid/issues/1#issuecomment-447051032 thus rendering it unusable.
@SimonLab created the Pull Request that would fix the failing test: https://github.com/nocursor/ex-cid/pull/2 and make the package useable and no response from the author.
As much as I would like to avoid duplicating effort, I/we (@dwyl & the wider Elixir community) cannot afford to rely on a package that has an unresponsive author. 😞
The cid
functionality is going to be at the core of all the code we will be writing for the next few years and if a bug (or security vulnerability) is discovered and we are unable to fix it because the "owner" of the package is AWOL, that will have a disastrous impact on our work.
I feel that this has left us with little choice but to re-do the work on cid
(and any dependencies) to ensure that the project is maintained.
Again, GOTO: #11
At present we are using a UUID in our Append-only Log see: https://github.com/dwyl/alog/issues/15 This is a good "stop gap" to get the
alog
project to "alpha" so it could be used in the Client Project. :shipit: But I feel that we should carefully consider the use of UUIDs as IDs in the "long term".If an ID is meant to be "machine readable" and "guaranteed" unique, then UUIDs are perfect. If there are additional requirements then we need to capture them.
For example: are we going to display the UUID in the URL for a given content type e.g:
Is this the most user-friendly ID we could display? (is it distinctive or memorable ...?)
Could we instantly improve UX by shortening the URL and making it Base64 instead of Base16? e.g:
Where the app would automatically
301
re-direct the request to:Can we "re-think" content/record IDs for both Uniqueness and Usability?
Basic Example: Address Book
The append-only log example: https://github.com/dwyl/phoenix-ecto-append-only-log-example demonstrated the benefits of using an append-only log for an address book. But we also explained that any app can benefit from having an immutable log as its' data store! see: https://github.com/dwyl/phoenix-ecto-append-only-log-example#examples-where-an-append-only-log-is-useful
What the example did not cover is how to mitigate against saving the same data multiple times. For example, consider the form:
if Bruce clicks/taps to
edit
his address, and we have an auto-save function to prevent loss of changes. We need a way of checking on the client that the address has not changedwhile
he is viewing the edit screen.Yes, this function should only be triggered by the "onChange" DOM event, but for argument's sake, we assume that Bruce clicks the "save" button (which we have discovered through UX-testing people still expect to have despite the "autosave"), should the server attempt to "re-save" the data that has not changed?
I suggest that by using a hash of the content as the Primary Key, Ecto (or PostgreSQL) would "reject" the insert request as a "duplicate" and we would not waste space in the database/table with dupe data.
If the data has not changed then the ContentID (
cid
) would be the same so no data insert.Use Case: Distributed Learning Platform
Learning systems universally have "vendor" or "platform" Lock-in. Do a lesson on Ask a question on StackOverflow? If your account gets banned for any reason, you lose "ownership" of any questions you had asked. Want to export your data and take it with you? Tough!
In our "product roadmap" we have begun to detail the creation of an Open Source Distributed Collaborative Learning platform: https://github.com/dwyl/product-roadmap/blob/master/collaborative-learning-community.md
By using a
cid
where each item of learning has a unique "fingerprint", _everyone learning on the platform can clearly see what each of their fellow learners has "covered". If anyone wants to either import or export their learning, they can do it easily.This is game-changing for making your "learning log" portable from kindergarten to the grave! No longer will your new school, university or employer/team have to rely on a "report card" or (incomplete) "transcript", everyone will be able to see that Alex
When Alex moves from Middle School to High School, all his teachers and classmates can easily see exactly what he has learned.
The teacher can be far more effective because they know what each student has already covered vs. what they still don't grasp. And instead of teaching more advanced topics that half the class will be confused by, they can attempt to cover the areas that still require work.
This is highly relevant in the workplace too! As a "hiring manager" or "team lead", I want to know exactly what my (potential) team members already know and what they are still trying to learn. I will delegate tasks to them that stretch their current abilities enough to encourage learning but not too much that would "overwhelm" them!
Closing the Gender/Class/Minority Divide in STEM/Tech
I feel that having a complete "learning log" will help to eliminate the gulf that exists in STEM/Tech because it will always be immediately obvious who has the knowledge/skill and it wont be a matter of which person has the loudest voice and most over-confidence.