MSX Metadata Hash update

[gr33k]

Ok - this one barely scraped any of the games. Many of these are Japanese, so I did my best to find the content. Most are complete with video and logo. A few are just logos. All of these are games mostly not programs (as there are a few programs I have as well that did not readily have logo/video available). Took quite some time to update these - Please let me know if you aren't able to pull them @thelamer :)

MSX: https://paste.thec0de.com/?43fc32a26d59834f#DfWTXjEhBMbHqQ2ZNsbeD2UjiU4LewCHzDWLwm2oerMU

[thelamer]

This is the only one that came through, the rest of them cannot be pulled from the network.

Write this to a file and exec it inside the emulatorjs container to pin these cids, this has all the ones you posted:

https://paste.thec0de.com/?742e86bc0de931a0#7s2RRW4RKn86TDfty7yyQ37RDc9puqu56o3tTwJTrq2B

docker exec -it emulatorjs bash > put the file in like /tmp or another directory > bash pinstuff.sh

[gr33k]

Thanks @thelamer - I did the command and went in to /tmp and created that pinstuff.sh file but got a couple of errors.

root@2a2c36a0efa2:/data/.ipfs/blocks# ipfs get QmbSUbmXQgVpko2Y9cPd85HSoHcVBMTDSsCaiiMEFDgbw3
Error: no IPFS repo found in /root/.ipfs.
please run: 'ipfs init'
root@2a2c36a0efa2:/data/.ipfs/blocks# ipfs init
generating ED25519 keypair...done
peer identity: 12D3KooWQuEb3WFzStD62nD5dJvMwHrZoJcuCyZZMUKTN9fAxp4A
initializing IPFS node at /root/.ipfs
root@2a2c36a0efa2:/data/.ipfs/blocks# cd /tmp
root@2a2c36a0efa2:/tmp# ls -l
total 316
-rwxr-xr-x 1 root root 321223 Sep 10 23:52 pinstuff.sh
root@2a2c36a0efa2:/tmp# ./pinstuff.sh
Error: block was not found locally (offline): ipld: could not find QmbSUbmXQgVpko2Y9cPd85HSoHcVBMTDSsCaiiMEFDgbw3
Error: block was not found locally (offline): ipld: could not find QmSYDPCngoassWAFjSTd4SBXDmMvYfbVKBSNN3qKKap9gV
Error: block was not found locally (offline): ipld: could not find QmV8not8tH2gZSKMEFWe6KxEmoBewQaibvfnHbhZC6uhG4
Error: block was not found locally (offline): ipld: could not find QmfD3f9zcz59B3VNMNvsLb2NEJHtF91xvyTm3K67XhDMri
Error: block was not found locally (offline): ipld: could not find QmfYhCGdofc9fux5F3VUsCBaJv43p2rj6yY338KeteyxAD
Error: block was not found locally (offline): ipld: could not find QmdXzP4cfUpsoM16NruoE1Le2CEiJRbK6VE77omSLSDDnX
Error: block was not found locally (offline): ipld: could not find QmUiHwwbPfKcMrYcg8XvbwqBVhTLkMtatVMYMQqpzBuMw6
Error: block was not found locally (offline): ipld: could not find QmYzca7dkPtag339jQ3b8Gp18cSxe91MReowFXEDDgxqCw
Error: block was not found locally (offline): ipld: could not find QmXiAuYjkqY5sKVjHdogJTTZ5odej7pnYdTQ4CmHAXv991
Error: block was not found locally (offline): ipld: could not find QmYXQbo5yTEeZZbAmxU3FkJ8QVrhWG16X8SZ2bMfVG2CHT

Not sure if this is part of the issue or if I am doing something wrong? I have rebuilt the server from scratch due to a raid failure...so I'm sad to see this isn't propagating automatically.

Happy to do whatever I have to do help share this stuff though!

UPDATE: I created a symlink in /root/.ipfs to /data/.ipfs and it seems to be doing stuff now:

root@2a2c36a0efa2:/tmp# ./pinstuff.sh
Saving file(s) to QmbSUbmXQgVpko2Y9cPd85HSoHcVBMTDSsCaiiMEFDgbw3
 749.11 KiB / 749.11 KiB [==========================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmSYDPCngoassWAFjSTd4SBXDmMvYfbVKBSNN3qKKap9gV
 73.66 KiB / 73.66 KiB [============================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmV8not8tH2gZSKMEFWe6KxEmoBewQaibvfnHbhZC6uhG4
 1.80 MiB / 1.80 MiB [==============================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmfD3f9zcz59B3VNMNvsLb2NEJHtF91xvyTm3K67XhDMri
 29.45 KiB / 29.45 KiB [============================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmfYhCGdofc9fux5F3VUsCBaJv43p2rj6yY338KeteyxAD
 420.66 KiB / 420.66 KiB [==========================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmdXzP4cfUpsoM16NruoE1Le2CEiJRbK6VE77omSLSDDnX
 41.07 KiB / 41.07 KiB [============================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmUiHwwbPfKcMrYcg8XvbwqBVhTLkMtatVMYMQqpzBuMw6
 489.53 KiB / 489.53 KiB [==========================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmYzca7dkPtag339jQ3b8Gp18cSxe91MReowFXEDDgxqCw
 22.82 KiB / 22.82 KiB [============================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmXiAuYjkqY5sKVjHdogJTTZ5odej7pnYdTQ4CmHAXv991
 636.95 KiB / 636.95 KiB [==========================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmYXQbo5yTEeZZbAmxU3FkJ8QVrhWG16X8SZ2bMfVG2CHT
 29.90 KiB / 29.90 KiB [============================================================================================================================================================================================================================================] 100.00% 0s
Saving file(s) to QmRwLVJacXwd1PLNpidfvSu21rDBehYtdz28BpkhsNyBFi
 634.35 KiB / 634.35 KiB [==========================================================================================================================================================================================================================================] 100.00% 0s

UPDATE 2: so it goes through a lot of files then eventually decides to hang after a given amount of them. I've tried removing the one in question but it just jams up on another one. When this happens - I can't even manually do ipfs get commands (from another terminal) they just sit there and hang. I'm wondering if there is some type of limit in ipfs that is getting reached? If so, how can I do this in chunks without filling up ipfs? Guessing here though - no errors happen it just gets stuck. From the list you provided - it gets stuck after QmcNs6hRCSG7PCdxYua9a9EX7P1rQh6BPgzd7hUDdRoXWL (Which successfully downloads then nothing else happens after). Still has plenty more to go through to complete the script :(

[thelamer]

FYI you can just set your home dir to /data

export HOME=/data

[gr33k]

Ok I made a smaller version of that script and I was able to put through almost 955 entries (get and add) without issues. I'm going to tinker with it some more to see if I can get the rest to go through even if I have to run 3 separate scripts. Even after running the shorter script - It still jams up when I run the second script...so there must be some kind of limit I'm hitting. I'll update the results here when I can..

UPDATE: restarted container and created 2nd part of that script to continue...but it still gets stuck after attempting 2 more then hangs

root@a18c4458b339:/tmp# ./test.sh
Saving file(s) to QmUZeB1VD6sRRtAva5BiPtzzoA25CYWc8jtYTo2bHpp9Gz
 991.29 KiB / 991.29 KiB [=================================================================================================] 100.00% 0s
Saving file(s) to QmcNs6hRCSG7PCdxYua9a9EX7P1rQh6BPgzd7hUDdRoXWL
 15.83 KiB / 15.83 KiB [===================================================================================================] 100.00% 0s

If there is some limit - I'm hitting it - do you have any suggestions? 😅

[thelamer]

My concern is and i have not confirmed it, is that the stuff is getting purged from your ipfs node because I set them to low power mode. You can add the files manually though just doing ipfs add * in the directory of files, this will pin them to your node.

[gr33k]

OK - it seems some of these files are 'bad'. Not sure why they get stuck, but if I remove them from the script or ctrl+c to cancel EACH of them (found over 100 so far in this situation), it will eventually continue with the ones that it can grab. I will then do ipfs add * in the directory and...VOILA - they should be done. Hopefully that means you will have them now?

Regarding the purge - yikes! - Something has to be going on though man. I have the port forwarded and traffic should be allowed (happy to check anything if ya need on my end). I'm happy to manually do whatever it takes to get these in to the cloud so others can benefit from a 'plug and play' experience (as well as me if I ever have to rebuild again lol).

Please let me know once you've confirmed that the files are being received and will not be purged (I know you said you haven't confirmed your suspicion as of yet). It would suck if the files are getting purged 😩

Thank you for your help as well!

[thelamer]

Well I was just saying you can go into the data directories and run ipfs add on the source files. This pull and push method is not needed you can technically refresh your node by adding the stuff in the folders. Does that make sense? Just have to do it from inside the container.

[gr33k]

Good news is I was able to pull the 'working' files from that list and added them.

Regarding the source files ... So I can go in to, for example, /data/arcade/videos or /data/arcade/logos and just run ipfs add * and that will work just the same?

Hell...that's easy - if that's really all I have to do, I'm happy to go in to each system and do it in all logo and video folders just in case 😄

[thelamer]

Yeah that's the ticket, you can do whole folders too, but don't want to pin the roms as well.

[gr33k]

Awesome! Ok, so I've gone into every single system I have setup and did ipfs add * explicitly in the /data/"system"/logo and video folder.

3do arcade atari2600 atari5200 atari7800 colecovision doom gb gba gbc jaguar lynx msx n64 nds nes ngp odyssey2 pce psx sega32x segaCD segaGG segaMD segaMS segaSG segaSaturn snes vb vectrex ws

Seems like at least 10GB+ worth of videos and logo files. If there is a way for you to confirm they have been received "in to the system" - please let me know 🍺

[thelamer]

Yes, pulling in now, it might take me a bit to merge all the json into master, but the pinning on the seed server is the hard part.

[thelamer]

These are in the latest release thanks @gr33k !

[gr33k]

Great news! It sounds like all the metadata tickets I've opened can be closed with a reference that we manually synced everything 😄 Thank you for your help @thelamer - you rock 🍺