fix: allow settings trust_remote_code for new huggingface version

[chanind]

Description

The newest version of Huggingface transformers won't allow loading old datasets that require allowing code execution without explicitly setting trust_remote_code=True. As a result, tests are now failing on CI since it loads the most recent version of transformers. This PR adds an option dataset_trust_remote_code which the user can set to allow this remote code execution so old datasets can be loaded in new versions of transformers, and also fixes CI.

Type of change

Please delete options that are not relevant.

• [x] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

Checklist:

• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [ ] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [ ] New and existing unit tests pass locally with my changes
• [x] I have not rewritten tests relating to key interfaces which would affect backward compatibility

You have tested formatting, typing and unit tests (acceptance tests not currently in use)

• [x] I have run make check-ci to check format and linting. (you can run make format to format code if needed.)

[jbloomAus]

@chanind Why make it optional None? We should just default to False?

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 59.68%. Comparing base (227d208) to head (e873ee9). Report is 1 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #187 +/- ## ========================================== + Coverage 59.59% 59.68% +0.09% ========================================== Files 25 25 Lines 2636 2642 +6 Branches 445 446 +1 ========================================== + Hits 1571 1577 +6 Misses 987 987 Partials 78 78 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[chanind]

I was worried that defaulting to False would be a breaking change, since then previously working code would no longer work if it's using an old dataset that requires setting the trust_remote_code variable. Setting it to None lets this be breaking or not based on Huggingface and not us.

[jbloomAus]

Ahhh I imagine setting to true avoids breaking change since if you trust remote code it will run?

On Mon, Jun 17, 2024, 10:58 PM David Chanin @.***> wrote:

I was worried that defaulting to False would be a breaking change, since then previously working code would no longer work if it's using an old dataset that requires setting the trust_remote_code variable. Setting it to None lets this be breaking or not based on Huggingface and not us.

— Reply to this email directly, view it on GitHub https://github.com/jbloomAus/SAELens/pull/187#issuecomment-2174502655, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQPMYZ7256KSKJW7L26NMUTZH5LYLAVCNFSM6AAAAABJNI2STWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZUGUYDENRVGU . You are receiving this because you modified the open/close state.Message ID: @.***>