Missing `Gemfile.lock` in repo

[alecdwm]

Because Gemfile.lock is in the .gitignore, it isn't included in this repo.

Without the lockfile, a fresh clone of this repo will always bundle install the latest available version of each gem.
This can lead to issues when breaking changes are released in one of the dependencies.

For example, the latest version of sqlite3-ruby has this change:

Raise ArgumentError if Database#execute, #execute_batch, or #query are passed multiple bind parameters that are not in an Array. In v2.0.0 these methods would silently swallow additional arguments, and this change makes the failure explicit.

Which causes this error when running notes_cloud_ripper.rb:

/usr/local/rvm/gems/default/gems/sqlite3-2.0.1/lib/sqlite3/database.rb:197:in `execute': wrong number of arguments (given 3, expected 1..2) (ArgumentError)
        from /workspaces/apple_cloud_notes_parser/lib/AppleNoteStore.rb:678:in `rip_notes'
        from /workspaces/apple_cloud_notes_parser/lib/AppleNoteStore.rb:221:in `rip_all_objects'
        from /workspaces/apple_cloud_notes_parser/lib/AppleBackup.rb:243:in `block in rip_notes'
        from /workspaces/apple_cloud_notes_parser/lib/AppleBackup.rb:240:in `each'
        from /workspaces/apple_cloud_notes_parser/lib/AppleBackup.rb:240:in `rip_notes'
        from notes_cloud_ripper.rb:194:in `<main>'

That specific error isn't too hard to fix, but there may be breaking changes in any of the other gems which don't necessarily cause the program to raise an error, but instead lead to works-on-my-machine-type bugs.
Also there might be more breaking changes in the future!

[threeplanetssoftware]

Thanks for flagging. The lack of checking in Gemfile.lock is somewhat intended as I know users were on different versions of Ruby and some gems, particularly Google's protobuf, have had their OWN breaking changes that required different versions in different versions of Ruby. I have not yet wanted to specify a Ruby version beyond "if it hasn't been end-of-lifed, it should work."

That said, I appreciate the heads up and will look at pushing a fix to this error with how execute is called.

[threeplanetssoftware]

The immediate issue of SQLite calls is addressed in a82014e01cf72bcb5852227d0fc08c5466f0356a. I will consider checking in a Gemfile that fits anything Ruby 3 and up.

[alecdwm]

Hey! Sounds good Great project btw 😀

There was another notable breaking change in the latest sqlite3-ruby:

Database#columns returns a list of internal frozen strings.

This causes AppleNote::is_gzip to raise a FrozenError when it tries to call data.force_encoding
Adding .dup should be all that's needed to fix it

-    return data.force_encoding("US-ASCII").start_with?("\x1F\x8B".force_encoding("US-ASCII"))
+    return data.dup.force_encoding("US-ASCII").start_with?("\x1F\x8B".force_encoding("US-ASCII"))

This didn't cause the program to crash, but it did mean that all exported notes had an empty content section.

---

After I ran into that error, I stopped looking into trying to fix any issues with the new deps and instead started to see if I could just configure a stricter Gemfile.

By adding these version ranges I was able to get everything working without any changes to the program source (also, I was using ruby 3.2!):

diff --git a/Gemfile b/Gemfile
index a485721..f681cdd 100644
--- a/Gemfile
+++ b/Gemfile
@@ -1,22 +1,22 @@
 source "https://rubygems.org"

 # Gems owned by Ruby directly
-gem 'cgi',">= 0.3.3" # Ruby official
-gem 'fileutils', ">= 1.4.0" # Ruby official
-gem 'openssl', ">= 2.2.0" # Ruby official
-gem 'rake', ">= 10.5.0" # Ruby official
-gem 'zlib', ">= 1.1.0" # Ruby official
+gem 'cgi',"~> 0.3" # Ruby official
+gem 'fileutils', "~> 1.4" # Ruby official
+gem 'openssl', "~> 2.2" # Ruby official
+gem 'rake', "~> 13.2" # Ruby official
+gem 'zlib', "~> 1.1" # Ruby official

 # Gems owned by Google
-gem 'google-protobuf', ">= 3.11.2" # Google official
+gem 'google-protobuf', "~> 4.26" # Google official

 # Gems owned by some "organization" on GitHub
 gem 'rspec' # Owned by rspec
-gem 'sqlite3', ">= 1.4.0" # Owned by SparkleMotion
+gem 'sqlite3', "~> 1.4" # Owned by SparkleMotion

 # Gems with multiple possible maintainers
-gem 'nokogiri', ">= 1.14.0" # Multiple maintainers
-gem 'keyed_archive', ">= 1.0.0" # Owned by me and one other
+gem 'nokogiri', "~> 1.14" # Multiple maintainers
+gem 'keyed_archive', "~> 1.0" # Owned by me and one other

 # Gems with just one maintainer
-gem 'aes_key_wrap', ">= 1.1.0" # Single owner: @tomdalling
+gem 'aes_key_wrap', "~> 1.1" # Single owner: @tomdalling

From what I could tell, ~> 1.1 is a shortcut for >= 1.1.0, < 2.0.0

[threeplanetssoftware]

Thanks for doing the hard work on that! I'm going to push make sure that Gemfile works with a few different versions on a clean installation and then push it, if so. It definitely will end support for Ruby 2.7 (which is end-of-lifed, I know, but there's possibly some other clean up to do for that). I want to make sure the database columns issue doesn't crop up anywhere else before I push that fix and call it good.

I appreciate the help!

[threeplanetssoftware]

Pushed the new Gemfile in e6d04752474d1bae9919367e4e7194c67a70d7e5 after building Docker images for Ruby 3.0 - 3.3 and testing it on a Mac.

[huyz]

The current Gemfile doesn't work for me on macOS 14.5 23F79 (Sonoma) MacBookPro18,2 (Apple M1 Max, arm64):

Details

``` ❯ bundle install Fetching gem metadata from https://rubygems.org/...... Resolving dependencies... Using rake 13.2.1 (was 13.1.0) Using bundler 2.3.7 Using diff-lcs 1.5.1 Using base64 0.2.0 Using nkf 0.2.0 Using rexml 3.2.6 Using racc 1.7.3 Using bigdecimal 3.1.8 Using fileutils 1.7.2 Using ipaddr 1.2.6 Using zlib 1.1.0 (was 3.1.0) Using CFPropertyList 3.0.7 Using aes_key_wrap 1.1.0 Using sqlite3 1.7.2 (arm64-darwin) Using keyed_archive 1.0.0 Using rspec-support 3.13.1 Using google-protobuf 4.27.0 (arm64-darwin) (was 3.25.3) Fetching openssl 2.2.3 (was 3.2.0) Using cgi 0.4.1 Using nokogiri 1.16.2 (arm64-darwin) Using rspec-core 3.13.0 Using rspec-expectations 3.13.0 Using rspec-mocks 3.13.0 Using rspec 3.13.0 Installing openssl 2.2.3 (was 3.2.0) with native extensions Gem::Ext::BuildError: ERROR: Failed to build gem native extension. current directory: /usr/local/git/apple_cloud_notes_parser/.direnv/ruby/gems/openssl-2.2.3/ext/openssl /usr/local/.rbenv/versions/3.1.2/bin/ruby -I /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0 -r ./siteconf20240529-18546-n9gs77.rb extconf.rb checking for t_open() in -lnsl... no checking for socket() in -lsocket... no checking for openssl/ssl.h... yes checking for CRYPTO_malloc() in -lcrypto... yes checking for SSL_new() in -lssl... yes checking for LIBRESSL_VERSION_NUMBER in openssl/opensslv.h... no checking for OpenSSL version >= 1.0.1 and < 3.0.0... no *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --without-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=/usr/local/.rbenv/versions/3.1.2/bin/$(RUBY_BASE_NAME) --with-openssl-dir --with-openssl-include --without-openssl-include=${openssl-dir}/include --with-openssl-lib --without-openssl-lib=${openssl-dir}/lib --with-kerberos-dir --without-kerberos-dir --with-kerberos-include --without-kerberos-include=${kerberos-dir}/include --with-kerberos-lib --without-kerberos-lib=${kerberos-dir}/lib --with-debug --without-debug --enable-debug --disable-debug --with-nsl-dir --without-nsl-dir --with-nsl-include --without-nsl-include=${nsl-dir}/include --with-nsl-lib --without-nsl-lib=${nsl-dir}/lib --with-nsllib --without-nsllib --with-socket-dir --without-socket-dir --with-socket-include --without-socket-include=${socket-dir}/include --with-socket-lib --without-socket-lib=${socket-dir}/lib --with-socketlib --without-socketlib --with-crypto-dir --without-crypto-dir --with-crypto-include --without-crypto-include=${crypto-dir}/include --with-crypto-lib --without-crypto-lib=${crypto-dir}/lib --with-cryptolib --without-cryptolib --with-ssl-dir --without-ssl-dir --with-ssl-include --without-ssl-include=${ssl-dir}/include --with-ssl-lib --without-ssl-lib=${ssl-dir}/lib --with-ssllib --without-ssllib extconf.rb:113:in `

': OpenSSL >= 1.0.1, < 3.0.0 or LibreSSL >= 2.5.0 is required (RuntimeError) To see why this extension failed to compile, please check the mkmf.log which can be found here: /usr/local/git/apple_cloud_notes_parser/.direnv/ruby/extensions/arm64-darwin-22/3.1.0/openssl-2.2.3/mkmf.log extconf failed, exit code 1 Gem files will remain installed in /usr/local/git/apple_cloud_notes_parser/.direnv/ruby/gems/openssl-2.2.3 for inspection. Results logged to /usr/local/git/apple_cloud_notes_parser/.direnv/ruby/extensions/arm64-darwin-22/3.1.0/openssl-2.2.3/gem_make.out /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/rubygems/ext/builder.rb:95:in `run' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:47:in `block in build' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/tempfile.rb:317:in `open' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:26:in `build' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/rubygems/ext/builder.rb:161:in `build_extension' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/rubygems/ext/builder.rb:195:in `block in build_extensions' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `each' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/rubygems/ext/builder.rb:192:in `build_extensions' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/rubygems/installer.rb:853:in `build_extensions' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/rubygems_gem_installer.rb:71:in `build_extensions' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/rubygems_gem_installer.rb:28:in `install' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/source/rubygems.rb:204:in `install' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:54:in `install' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/installer/gem_installer.rb:16:in `install_from_spec' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:186:in `do_install' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/installer/parallel_installer.rb:177:in `block in worker_pool' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/worker.rb:62:in `apply_func' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/worker.rb:57:in `block in process_queue' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/worker.rb:54:in `loop' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/worker.rb:54:in `process_queue' /usr/local/.rbenv/versions/3.1.2/lib/ruby/3.1.0/bundler/worker.rb:91:in `block (2 levels) in create_threads' An error occurred while installing openssl (2.2.3), and Bundler cannot continue. In Gemfile: openssl ```

If I change part of Gemfile back to:

# Gems owned by Ruby directly
gem 'cgi',"~> 0.3" # Ruby official
gem 'fileutils', "~> 1.4" # Ruby official
gem 'openssl', ">= 2.2.0" # Ruby official
[…]

`

then installation works again:

Details

``` ❯ bundle install Fetching gem metadata from https://rubygems.org/...... Resolving dependencies... Using rake 13.2.1 (was 13.1.0) Using bundler 2.3.7 Using diff-lcs 1.5.1 Using racc 1.7.3 Using nkf 0.2.0 Using rexml 3.2.6 Using aes_key_wrap 1.1.0 Using rspec-support 3.13.1 Using sqlite3 1.7.2 (arm64-darwin) Using zlib 1.1.0 (was 3.1.0) Using base64 0.2.0 Using openssl 3.2.0 Using bigdecimal 3.1.8 Using fileutils 1.7.2 Using rspec-expectations 3.13.0 Using nokogiri 1.16.2 (arm64-darwin) Using rspec-core 3.13.0 Using google-protobuf 4.27.0 (arm64-darwin) (was 3.25.3) Using CFPropertyList 3.0.7 Using cgi 0.4.1 Using rspec-mocks 3.13.0 Using keyed_archive 1.0.0 Using rspec 3.13.0 Bundle complete! 11 Gemfile dependencies, 23 gems now installed. Use `bundle info [gemname]` to see where a bundled gem is installed. ```