Add Cargo.lock file

[malob]

Add Cargo.lock since this is best practice for the crate-type used in this package.

From The Cargo Book section Cargo.toml vs Cargo.lock:

If you’re building an end product, which are executable like command-line tool or an application, or a system library with crate-type of staticlib or cdylib, check Cargo.lock into git.

[hauntsaninja]

tiktoken is none of the things you mention in your quote

[malob]

Is it not?

https://github.com/openai/tiktoken/blob/82facf911fe41f39f13565be73899eab1e1761ef/Cargo.toml#L9

From The Rust Reference section Linkage:

--crate-type=cdylib, #![crate_type = "cdylib"] - A dynamic system library will be produced. This is used when compiling a dynamic library to be loaded from another language. This output type will create *.so files on Linux, *.dylib files on macOS, and *.dll files on Windows.

[malob]

Seems to me like tiktoken is a Python library that depends on a Rust dynamic system library of crate_type = "cdylib".

[hauntsaninja]

It's just a library, not a system library. The rust code isn't an end product. The lock file isn't solving any problems here. If you're looking to redistribute the tiktoken rust code as some sort of end product, you should lock it yourself.