`DashboardRunner` errors

[joelburget]

Describe the bug

Attempt 1 (https://github.com/joelburget/SAELens/blob/1f59808ca9fd76e0e946ea6e631c65a4f8482240/eval_polar.py):

runner = DashboardRunner(sae_path=sae_path)
runner.run()

Result:

Forward passes to cache data for vis: 100%|█████████████████████████████████████████████████████████████████████████████████| 24/24 [03:04<00:00,  7.71s/it]
Extracting vis data from cached data: 100%|█████████████████████████████████████████████████████████████████████████████| 1024/1024 [03:04<00:00,  5.54it/s]
  0%|                                                                                                                                 | 0/8 [03:05<?, ?it/s]
Traceback (most recent call last):
  File "/workspace/SAELens/eval_polar.py", line 16, in <module>
    runner.run()
  File "/workspace/SAELens/sae_lens/analysis/dashboard_runner.py", line 376, in run
    feature_data.save_feature_centric_vis(
  File "/workspace/SAELens/.venv/lib/python3.10/site-packages/sae_vis/data_storing_fns.py", line 1073, in save_feature_centric_vis
    assert self.model is not None
AssertionError

This hack (https://github.com/joelburget/SAELens/commit/08bddf00cbacf4da919340fc783c51fc68ab73fd) seems to fix the immediate issue.

Attempt 2 (with hack) result:

Forward passes to cache data for vis: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 24/24 [03:10<00:00,  7.92s/it]
Extracting vis data from cached data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1024/1024 [03:10<00:00,  5.39it/s]
  0%|                                                                                                                                                                                 | 0/8 [03:10<?, ?it/s]
Traceback (most recent call last):
  File "/workspace/SAELens/eval_polar.py", line 15, in <module>
    runner.run()
  File "/workspace/SAELens/sae_lens/analysis/dashboard_runner.py", line 376, in run
    feature_data.save_feature_centric_vis(
  File "/workspace/SAELens/.venv/lib/python3.10/site-packages/sae_vis/data_storing_fns.py", line 1085, in save_feature_centric_vis
    html_obj = feature_data._get_html_data_feature_centric(
  File "/workspace/SAELens/.venv/lib/python3.10/site-packages/sae_vis/data_storing_fns.py", line 860, in _get_html_data_feature_centric
    html_obj += component._get_html_data(
  File "/workspace/SAELens/.venv/lib/python3.10/site-packages/sae_vis/data_storing_fns.py", line 259, in _get_html_data
    assert cfg.n_rows <= len(self.bottom_logits)
AssertionError

System Info I'm running within that SAELens repo, dependencies installed via poetry.

(sae-lens-py3.10) root@e8e8a5a21111:/workspace/SAELens# python3 --version
Python 3.10.12
(sae-lens-py3.10) root@e8e8a5a21111:/workspace/SAELens# uname -a
Linux e8e8a5a21111 5.15.0-89-generic #99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Additional context It would be helpful to see an example of how to use DashboardRunner. It would also be useful to see an example of how to use the wandb integration (https://github.com/joelburget/SAELens/commit/1f59808ca9fd76e0e946ea6e631c65a4f8482240).

Checklist

• [x] I have checked that there is no similar issue in the repo (seems unrelated to https://github.com/jbloomAus/SAELens/issues/72)

[jbloomAus]

Hey Joel,

Sorry for those issues and the delay in responding. For various reasons the dashboard runner I wrote a few months ago became stale fairly quickly but I think the SAE Vis demo works well for my residual stream SAEs. https://github.com/callummcdougall/sae_vis/blob/main/demo.ipynb.

If you have trouble with this let me know / feel free to reach out to me or @chanind on Slack.

[joelburget]

Thanks @jbloomAus, I think this gets me unstuck. Feel free to close this ticket (or leave it open as a reminder to update or remove DashboardRunner).

[jbloomAus]

Thanks, will do! Please let me know you get on. Would love to make dashboard gen simpler/faster in the near future.