"No such file or directory: 'generator/gpt2/models/model_v5/vocab.bpe'"

[JeffreyBenjaminBrown]

When I go to this page and select "Run all", this is what I get:

AI Dungeon 2 will save and use your actions and game to continually improve AI
 Dungeon. If you would like to disable this enter 'nosaving' for any action.
 This will also turn off the ability to save games.

Initializing AI Dungeon! (This might take a few minutes)

Traceback (most recent call last):
  File "play.py", line 211, in <module>
    play_aidungeon_2()
  File "play.py", line 74, in play_aidungeon_2
    generator = GPT2Generator()
  File "/content/AIDungeon/AIDungeon/AIDungeon/AIDungeon/AIDungeon/generator/gpt2/gpt2_generator.py", line 27, in __init__
    self.enc = encoder.get_encoder(self.model_name, models_dir)
  File "/content/AIDungeon/AIDungeon/AIDungeon/AIDungeon/AIDungeon/generator/gpt2/src/encoder.py", line 111, in get_encoder
    with open(os.path.join(models_dir, model_name, 'vocab.bpe'), 'r', encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'generator/gpt2/models/model_v5/vocab.bpe'

[skryspin]

Yes, I also am getting that error.

[thelittlejunglemirza]

I was getting the same error and I used a different sign in now gives me this error:

Traceback (most recent call last):
  File "play_dm.py", line 48, in <module>
    play_dm()
  File "play_dm.py", line 21, in play_dm
    generator = GPT2Generator(temperature=0.9)
  File "/content/AIDungeon/generator/gpt2/gpt2_generator.py", line 27, in __init__
    self.enc = encoder.get_encoder(self.model_name, models_dir)
  File "/content/AIDungeon/generator/gpt2/src/encoder.py", line 109, in get_encoder
    with open(os.path.join(models_dir, model_name, 'encoder.json'), 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'generator/gpt2/models/model_v5/encoder.json'```

[azackoff2]

I am getting that error as well. That file definitely doesn't exist in the directory location that is specified.

[Jonblu11]

I am also getting the error.

Initializing AI Dungeon! (This might take a few minutes)

Traceback (most recent call last):
  File "play.py", line 211, in <module>
    play_aidungeon_2()
  File "play.py", line 74, in play_aidungeon_2
    generator = GPT2Generator()
  File "/content/AIDungeon/AIDungeon/generator/gpt2/gpt2_generator.py", line 27, in __init__
    self.enc = encoder.get_encoder(self.model_name, models_dir)
  File "/content/AIDungeon/AIDungeon/generator/gpt2/src/encoder.py", line 111, in get_encoder
    with open(os.path.join(models_dir, model_name, 'vocab.bpe'), 'r', encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'generator/gpt2/models/model_v5/vocab.bpe'

[Autistic-Unicorn]

`12/08 01:51:06 [NOTICE] Downloading 1 item(s)

12/08 01:51:06 [ERROR] CUID#7 - Download aborted. URI=http://**/model_v5/model-550.data-00000-of-00001 Exception: [AbstractCommand.cc:351] errorCode=22 URI=http://**/model_v5/model-550.data-00000-of-00001 -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=403

12/08 01:51:06 [NOTICE] Download GID#8f01c46017c97048 not complete:

Download Results: gid |stat|avg speed |path/URI ======+====+===========+======================================================= 8f01c4|ERR | 0B/s|http://**/model_v5/model-550.data-00000-of-00001

Status Legend: (ERR):error occurred.

aria2 will resume download if the transfer is restarted. If there are any errors, then see the log file. See '-l' option in help/man page for details. --2019-12-08 01:51:06-- http://**/model_v5/checkpoint Connecting to 130.211.31.182:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-12-08 01:51:06 ERROR 403: Forbidden.

--2019-12-08 01:51:06-- http://**/model_v5/encoder.json Connecting to 130.211.31.182:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-12-08 01:51:06 ERROR 403: Forbidden.

--2019-12-08 01:51:06-- http://**/model_v5/hparams.json Connecting to 130.211.31.182:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-12-08 01:51:07 ERROR 403: Forbidden.

--2019-12-08 01:51:07-- http://**/model_v5/model-550.index Connecting to 130.211.31.182:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-12-08 01:51:07 ERROR 403: Forbidden.

--2019-12-08 01:51:07-- http://**/model_v5/model-550.meta Connecting to 130.211.31.182:80... connected. HTTP request sent, awaiting response... 200 OK Length: 14316502 (14M) [application/octet-stream] Saving to: ‘model-550.meta’

model-550.meta 100%[===================>] 13.65M --.-KB/s in 0.07s

2019-12-08 01:51:07 (190 MB/s) - ‘model-550.meta’ saved [14316502/14316502]

--2019-12-08 01:51:07-- http://**/model_v5/vocab.bpe Connecting to 130.211.31.182:80... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-12-08 01:51:07 ERROR 403: Forbidden.``

There is 403 error while downloading and this causes errors while initializing the AI Dungeon. I "**" the ip adresses since i don't know if it save to share them.

[ghost]

you beat me to it - I think they probably turned off downloading the necessary models. How big are they?

[ghost]

Same here

[ghost]

Looks like Nick updated the download URL - getting "Network is unreachable" now

Anyone able to set up a mirror?

[ghost]

@nickwalton how big are the files that need to be hosted?

[JillCrungus]

A recent commit changed the download url in the install.sh file to an IP that returns 403 errors.

A commit that was just pushed changed the URL again to https://students.cs.byu.edu/~nickwalt but this site doesn't work either.

You can temporarily hotfix it on your end by changing the download_url back to the original, https://aidungeonmodel.s3-us-west-1.amazonaws.com until Nick can sort it out

[ghost]

Files don't seem that big, why not just host them on the git repo?

[nickwalton]

I think I just fixed this issue. Sorry for the craziness guys. It was just costing INSANE amounts. But should be reasonable now. Let me know if it still doesn't work.

[ghost]

ahh, the apt-get install of aria2 failed because running locally it needed sudo. the errors weren't visible when running the install script.

I see now that its 5.8 GiB

This is downloaded everytime someone runs this in that notebook, or is there caching? Sorry still downloading right now locally to my machine, haven't retried the notebook.

[JeffreyBenjaminBrown]

@nickwalton -- Details, please! What was the cost per game before the fix? The cost now? How many people are playing it?

[JeffreyBenjaminBrown]

And is it your personal money, or the university's?

[ghost]

I'm going to make an issue for this problem

[ghost]

I think I just fixed this issue. Sorry for the craziness guys. It was just costing INSANE amounts. But should be reasonable now. Let me know if it still doesn't work.

Cloning into 'AIDungeon'... remote: Enumerating objects: 48, done. remote: Counting objects: 100% (48/48), done. remote: Compressing objects: 100% (39/39), done. remote: Total 48 (delta 2), reused 22 (delta 0), pack-reused 0 Unpacking objects: 100% (48/48), done. /content/AIDungeon Downloading AIDungeon2 Model... (this may take a few minutes)

12/08 03:34:37 [NOTICE] Downloading 1 item(s)

12/08 03:34:37 [ERROR] CUID#13 - Download aborted. URI=http://130.211.31.182:80/model_v5/model-550.data-00000-of-00001 Exception: [AbstractCommand.cc:351] errorCode=22 URI=http://130.211.31.182:80/model_v5/model-550.data-00000-of-00001 -> [HttpSkipResponseCommand.cc:239] errorCode=22 The response status is not successful. status=403

Additionally, the workaround now fails with: https://aidungeonmodel.s3-us-west-1.amazonaws.com/model_v5/hparms.json returning:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>A1CC78CD07CD6853</RequestId>
<HostId>
8Coije0L8KTLMiWRGinwdYfzmmc6YzwAF/zrvtNxdtC1RlJg02MY2BVP5iRIWJqBGyMeB66jo+g=
</HostId>
</Error>

[anonymousUsers1]

Running into the same issue, resulting in the following message:

`AI Dungeon 2 will save and use your actions and game to continually improve AI Dungeon. If you would like to disable this enter 'nosaving' for any action. This will also turn off the ability to save games.

Initializing AI Dungeon! (This might take a few minutes)

Traceback (most recent call last): File "play.py", line 211, in play_aidungeon_2() File "play.py", line 74, in play_aidungeon_2 generator = GPT2Generator() File "/content/AIDungeon/AIDungeon/generator/gpt2/gpt2_generator.py", line 29, in init with open(os.path.join(models_dir, self.model_name, 'hparams.json')) as f: FileNotFoundError: [Errno 2] No such file or directory: 'generator/gpt2/models/model_v5/hparams.json'`

[ghost]

@nickwalton -- Details, please! What was the cost per game before the fix? The cost now? How many people are playing it?

13.34¢ USD per run with current S3 prices, which doesn't sound like a lot, but according to Google, searches for "AI Dungeon 2" are quickly coming close to eclipsing searches for "World of Warcraft" to give you an idea of how much traffic this is getting. It's been republished by a ton of large tech and gaming sites/blogs/news aggregators.

[anonymousUsers1]

It's been republished by a ton of large tech and gaming sites/blogs/news aggregators.

The futurism.com article about this project appeared on my google chrome home screen a few hours ago, so that may lead to a larger influx soon.

[nickwalton]

Turns out fix wasn't good enough. To answer your questions cost was something like 20-30 cents per download. Got 60,000 unique users to the aidungeon.io site and the charges ended up at 15k for just today. Dr. Wingate, the professor of my lab was sponsoring it, but it's gotten past what he can afford from the labs budget so I had to shut off public bucket access till there is a solution. If anyone wants to set up a torrent system or something then I'm happy to support

[snglth]

IPFS may be a way to handle the model distribution

[JeffreyBenjaminBrown]

Omg. You guys are amazing.

What's a run? If someone selects "restart and run all", is that a new run? I may have done that 40 times today.

[ghost]

i'd gladly help seed any torrents!

[snglth]

Torrents should be fine too. You can feed a magnet link to aria2c to download files in colab. Someone who already has model files just need to register torrent file on some tracker.

[ghost]

i'd gladly help seed any torrents!

Same, though I suspect Nick probably doesn't want to spend the time re-implementing this download over a bittorrent client inside of a shell script.

If it's any consideration, I know OVH does storage at 1.1¢ per GB, which works out to only about 6.38¢/run, which would decrease your costs significantly, but at the levels of bandwidth we're talking about, I'm thinking the most cost effective option would be to host the game as it's own service on a dedicated server with plenty of processing power.

[ghost]

@InfosecRD see above, aria2c can do magnet links

[ghost]

@InfosecRD see above, aria2c can do magnet links

I'm no expert on the colab platform, but if this is implemented, I'll help seed from my home connection and on my VPS :)

[nickwalton]

Yeah my current focus is getting a hosted solution up. It'll be a cleaner experience for maybe even cheaper. But if anyone wants to implement torrent download install ill gladly merge it in

On Sat, Dec 7, 2019, 9:24 PM . notifications@github.com wrote:

@InfosecRD https://github.com/InfosecRD see above, aria2c can do magnet links

I'm no expert on the colab platform, but if this is implemented, I'll help seed locally and on my VPS :)

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/nickwalton/AIDungeon/issues/39?email_source=notifications&email_token=AFJNOQHBFSPCU3QCOUZENC3QXRZIBA5CNFSM4JXU5OC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGGVT4I#issuecomment-562911729, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFJNOQC6B72MEVVIHA7EIBLQXRZIBANCNFSM4JXU5OCQ .

[ghost]

yeah aria2c can just take a magnet:// link as an input but that requires someone with the files to actually start hosting the torrents in the first place

[stylemistake]

I am in process of mirroring contents of the university website, and will put up a torrent in 30 minutes or so (uni bandwidth is really low).

[JeffreyBenjaminBrown]

Does the surge in traffic coincide with AI Weirdness's post yesterday?

This game is astounding, literally groundbreaking. How do your profs feel about you dropping out for a couple years to get rich on the idea before someone else does?

[stylemistake]

Torrent is up, help seeding.

Torrent file: model_v5.torrent.zip

Magnet:

magnet:?xt=urn:btih:b343b83b35bff774dab13e0281ce13b3daf37d3e&dn=model%5Fv5&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.pomf.se%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fdenis.stalker.upeer.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cyberia.is%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker3.itzmx.com%3A6961%2Fannounce&tr=udp%3A%2F%2Fzephir.monocul.us%3A6969%2Fannounce&tr=udp%3A%2F%2Fxxxtor.com%3A2710%2Fannounce

Feel free to remove some of the tracker information to make this magnet more compact.

[ghost]

gonna seed it with my VPS but I'll have to cut it off before it reaches it's own bandwidth quota lol

[Portaluke]

Honestly, I've never used a Torrent before, but hopefully I've done everything I needed to to help. (If anyone had any tips for me in the meantime that would be appreciated) Onto seeding now, hope this helps people.

[p-groarke]

You could host the files on a free google drive account (15GB), and use a link generator to get a direct link to it. Will seed torrent for now. Good luck!

[nickwalton]

Awesome guys! And I have one week till I graduate so might as well just finish haha. Though my GPA this semester is a dumpster fire from working on this so much.

On Sat, Dec 7, 2019, 11:05 PM p-groarke notifications@github.com wrote:

You could host the files on a free google drive account (15GB), and use a link generator to get a direct link to it. Will seed torrent for now. Good luck!

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/nickwalton/AIDungeon/issues/39?email_source=notifications&email_token=AFJNOQHRJW4MRTOXYPCXN63QXSFEBA5CNFSM4JXU5OC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGGWWMY#issuecomment-562916147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFJNOQDPCBHMTSGHEMML6GLQXSFEBANCNFSM4JXU5OCQ .

[Akababa]

I've got the folder shared on my google drive too, this way you can mount the file directly to avoid copying: https://drive.google.com/drive/folders/1XiDD2BD8vLZaJxZpCrNYjscpvnD3EYrP

[stylemistake]

mount the file directly to avoid copying

Brilliant idea, Nick might in fact use his own GDrive to have more control over his models.

[Portaluke]

mount the file directly to avoid copying

Brilliant idea, Nick might in fact use his own GDrive to have more control over his models.

Out of curiosity, would this make the torrent obsolete as far as the progress we made on it?

[nickwalton]

Wait how would this work? Can't people only mount their own Google drive?

On Sat, Dec 7, 2019, 11:34 PM Portaluke notifications@github.com wrote:

mount the file directly to avoid copying

Brilliant idea, Nick might in fact use his own GDrive to have more control over his models.

Out of curiosity, would this make the torrent obsolete as far as the progress we made on it?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/nickwalton/AIDungeon/issues/39?email_source=notifications&email_token=AFJNOQDEH7STIBHXOCOEEBTQXSINXA5CNFSM4JXU5OC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGGXA3I#issuecomment-562917485, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFJNOQD73FF4P6YIIAPAZBLQXSINXANCNFSM4JXU5OCQ .

[Akababa]

They'd have to click the link manually first to get it as a shared folder under their drive, and then they mount their own drive. https://stackoverflow.com/questions/53576555/share-a-part-of-google-drive-on-colab

[JushBJJ]

Cloning into 'AIDungeon'...
remote: Enumerating objects: 48, done.
remote: Counting objects: 100% (48/48), done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 48 (delta 2), reused 20 (delta 0), pack-reused 0
Unpacking objects: 100% (48/48), done.
/content/AIDungeon/AIDungeon
Downloading AIDungeon2 Model... (this may take a few minutes)
--2019-12-08 06:48:35--  https://github.com/nickwalton/AIDungeon/files/3935881/model_v5.torrent.zip
Resolving github.com (github.com)... 140.82.114.4
Connecting to github.com (github.com)|140.82.114.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-repository-file-5c1aeb.s3.amazonaws.com/179196443/3935881?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191208%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191208T064836Z&X-Amz-Expires=300&X-Amz-Signature=66861f1a4a2ae662d4bfd35f90081cf799b4a456004efe3bebbefd32c2135cb9&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3Bfilename%3Dmodel_v5.torrent.zip&response-content-type=application%2Fzip [following]
--2019-12-08 06:48:36--  https://github-production-repository-file-5c1aeb.s3.amazonaws.com/179196443/3935881?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20191208%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20191208T064836Z&X-Amz-Expires=300&X-Amz-Signature=66861f1a4a2ae662d4bfd35f90081cf799b4a456004efe3bebbefd32c2135cb9&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3Bfilename%3Dmodel_v5.torrent.zip&response-content-type=application%2Fzip
Resolving github-production-repository-file-5c1aeb.s3.amazonaws.com (github-production-repository-file-5c1aeb.s3.amazonaws.com)... 52.217.37.204
Connecting to github-production-repository-file-5c1aeb.s3.amazonaws.com (github-production-repository-file-5c1aeb.s3.amazonaws.com)|52.217.37.204|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 60573 (59K) [application/zip]
Saving to: ‘model_v5.torrent.zip’

model_v5.torrent.zi 100%[===================>]  59.15K   162KB/s    in 0.4s    

2019-12-08 06:48:37 (162 KB/s) - ‘model_v5.torrent.zip’ saved [60573/60573]

Archive:  model_v5.torrent.zip
  inflating: model_v5.torrent        

12/08 06:48:37 [NOTICE] Downloading 1 item(s)

12/08 06:48:37 [ERROR] Exception caught while loading DHT routing table from /root/.cache/aria2/dht.dat
Exception: [DHTRoutingTableDeserializer.cc:83] errorCode=1 Failed to load DHT routing table from /root/.cache/aria2/dht.dat

12/08 06:48:37 [NOTICE] IPv4 DHT: listening on UDP port 6926

12/08 06:48:37 [NOTICE] IPv4 BitTorrent: listening on TCP port 6900

12/08 06:48:37 [ERROR] IPv6 BitTorrent: failed to bind TCP port 6900
Exception: [SocketCore.cc:312] errorCode=1 Failed to bind a socket, cause: Name or service not known

I did this in the colab website.

[stylemistake]

This error is misleading, because it simply fails to bind to IPv6 port, but it was downloading fine over IPv4.

[ghost]

I think the Google drive idea might be better in practice

[Akababa]

I'm working on the Google drive symlink workaround, should be ready soon.

[WAUthethird]

Google Drive workaround: https://colab.research.google.com/drive/1OjBQe4H4C2s-p4-OeJoXw5DStIjPy2VS

[Akababa]

Google Drive workaround: https://colab.research.google.com/drive/1OjBQe4H4C2s-p4-OeJoXw5DStIjPy2VS

Why does it copy the files? Doesn't the shared folder work fine?

[WAUthethird]

Google Drive workaround: https://colab.research.google.com/drive/1OjBQe4H4C2s-p4-OeJoXw5DStIjPy2VS

Why does it copy the files? Doesn't the shared folder work fine?

No, because this method requires mounting the user's Google Drive.

[Akababa]

@WAUthethird So I don't quite understand. Why would you download from your google drive instead of mounting your google drive?