Feature Request: Bulk Upload Amiibo

[Cabalist]

Neat project. :) Poking around I found I could only upload one amiibo at a time. Are there plans to be able to upload them by directory?

Thanks!

[Xerxes3rd]

Thanks! I’d like to have zip file uploads. There’s a small zip/unzip library I found, but haven’t tried integrating it yet.

[Cabalist]

I am not as familiar with the ESP8266 libraries you are using but would adding multiple to the file upload field be handled? (i.e. https://www.w3schools.com/tags/att_input_multiple.asp)

[Cabalist]

And I guess allowing uploadAmiibo to take an iterable instead of a single file

[Xerxes3rd]

I seem to recall looking at this and it wasn't supported on the ESP side. I'm using ESPAsyncWebServer for this portion. I'll take another look at it.

BTW, pretty much any JavaScript/HTML fixes and/or layout enhancements are welcome. I'm sure it's obvious that I'm not a frontend developer, as it took me a lot of time & hacking just to get what I have working (and the JavaScript is pretty ugly).

[Cabalist]

Mannnnn... me either. haha. I'll keep poking around I am pretty unfamiliar with this toolchain. The fixes I did were pretty much my linter setup blinking at me and not some great feat on my part.

It does look like ESPAsyncWebServer fixed something particular to multiple file uploads: https://github.com/me-no-dev/ESPAsyncWebServer/issues/76 https://github.com/me-no-dev/ESPAsyncWebServer/pull/104/files

This was over a year ago though. At least it means they support it. ;)

[Xerxes3rd]

That's good to hear. I'll have to re-work how uploads work, since right now you upload a single amiibo, then it shows you the data of the one you just uploaded. It might make more sense to have the upload portion on the main page (and not on a "popup") that then just updates the main list as it decodes amiibos.

The same usage pattern should work with a multi-file upload or a single zip file (assuming I can get that working). There probably should be some sort of message letting you know if you upload an invalid file as well, but I haven't figured out a clean way of displaying such things.

[Xerxes3rd]

After further consideration, bulk uploads/downloads should be handled mostly in the browser using a zip/unzip JavaScript library. With the memory constraints on the ESP, it would be difficult or impossible to handle a zip file of any significant size.

[Cabalist]

I don't believe this belongs in the main repo but here is how I solved needing to upload a bunch of data files:

Pass the IP address of your wifiibo and the data dir with amiibos in the top level.

Also neat: You can even watch the progress on the "Upload Amiibo" page since wifiibo uses websockets.

# coding=utf-8
from __future__ import print_function

import argparse
import os
import sys
import time

import requests

def batch_upload_amiibos(wifiibo_ip, data_dir):
    amiibo_bins_to_upload = [name for name in os.listdir(data_dir) if os.path.isfile(os.path.join(data_dir, name)) and ".bin" in name]
    amiibo_count = len(amiibo_bins_to_upload)

    dest_url = 'http://{}/uploadAmiibo'.format(wifiibo_ip)

    item_count = 0
    for each_item in amiibo_bins_to_upload:
        if ".bin" not in each_item:
            pass

        item_count += 1
        print("Working on {} of {}\r".format(item_count, amiibo_count), file=sys.stdout, end=" ")
        fp = os.path.join(data_dir, each_item)
        requests.post(dest_url, files={'data': (os.path.split(fp)[1], open(fp, "rb"))})
        time.sleep(1)
    print("Done! Uploaded {} amiibos!".format(amiibo_count), file=sys.stdout)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("ip", help="IP address of wifiibo [i.e. 192.168.4.1]", type=str)
    parser.add_argument("amiibo_dir", help="Directory with amiibo bin files", type=str)
    args = parser.parse_args()

    batch_upload_amiibos(args.ip, os.path.normpath(args.amiibo_dir))

[Xerxes3rd]

Awesome! I could see possibly putting the script in a "tools" folder or something like that. BTW, for each file Wifiibo finds that ends in .bin, Wifiibo does a sanity size check on it too, which keeps it from trying to read keys_retail.bin as an amiibo. The sizes are defined in /src/amiibo.h, and are:

#define NFC3D_AMIIBO_SIZE_SMALL 532
#define NFC3D_AMIIBO_SIZE_HASH 572

The #define naming isn't ideal, but it should cover all the dumps I've encountered so far.

I could see expanding the script to also do a bulk download by doing the repeated "listamiibo_chunk" function like the JS does, then grabbing each file. That would be a really nice stopgap until full bulk upload/download gets integrated into the main application.

[StanMOI]

I don't believe this belongs in the main repo but here is how I solved needing to upload a bunch of data files:

Pass the IP address of your wifiibo and the data dir with amiibos in the top level.

Also neat: You can even watch the progress on the "Upload Amiibo" page since wifiibo uses websockets.

# coding=utf-8
from __future__ import print_function

import argparse
import os
import sys
import time

import requests

def batch_upload_amiibos(wifiibo_ip, data_dir):
    amiibo_bins_to_upload = [name for name in os.listdir(data_dir) if os.path.isfile(os.path.join(data_dir, name)) and ".bin" in name]
    amiibo_count = len(amiibo_bins_to_upload)

    dest_url = 'http://{}/uploadAmiibo'.format(wifiibo_ip)

    item_count = 0
    for each_item in amiibo_bins_to_upload:
        if ".bin" not in each_item:
            pass

        item_count += 1
        print("Working on {} of {}\r".format(item_count, amiibo_count), file=sys.stdout, end=" ")
        fp = os.path.join(data_dir, each_item)
        requests.post(dest_url, files={'data': (os.path.split(fp)[1], open(fp, "rb"))})
        time.sleep(1)
    print("Done! Uploaded {} amiibos!".format(amiibo_count), file=sys.stdout)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("ip", help="IP address of wifiibo [i.e. 192.168.4.1]", type=str)
    parser.add_argument("amiibo_dir", help="Directory with amiibo bin files", type=str)
    args = parser.parse_args()

    batch_upload_amiibos(args.ip, os.path.normpath(args.amiibo_dir))

What file a have to edit to make it works?

[StanMOI]

in readme: ...you can place existing amiibo dumps and the retail key file into the sketch Data directory, and use the "ESP8266 Sketch Data Upload" option in the "Tools" menu to upload them all at once...

I don't find it "ESP8266 Sketch Data Upload"

[877dev]

in readme: ...you can place existing amiibo dumps and the retail key file into the sketch Data directory, and use the "ESP8266 Sketch Data Upload" option in the "Tools" menu to upload them all at once...

I don't find it "ESP8266 Sketch Data Upload"

If you google “ESP8266 Sketch Data Upload” there’s lots of results showing how to do that. If you can’t figure that out maybe need to do some reading 👍

[StanMOI]

in readme: ...you can place existing amiibo dumps and the retail key file into the sketch Data directory, and use the "ESP8266 Sketch Data Upload" option in the "Tools" menu to upload them all at once... I don't find it "ESP8266 Sketch Data Upload"

If you google “ESP8266 Sketch Data Upload” there’s lots of results showing how to do that. If you can’t figure that out maybe need to do some reading 👍

Hi, I googled... Well, I try it and done some tests. Its works but with no images :(

[Xerxes3rd]

Wifiibo pulls amiibo pictures from the Internet, so you won't see them if the Wifiibo isn't connected to the Internet. Additionally, it won't show any amiibo dumps if the decryption keys aren't loaded.

[StanMOI]

Wifiibo pulls amiibo pictures from the Internet, so you won't see them if the Wifiibo isn't connected to the Internet. Additionally, it won't show any amiibo dumps if the decryption keys aren't loaded.

Thanks. It works but only when i upload by one file. If i use folder upload, i have no pictures.

[Xerxes3rd]

Folders are not supported.

[Xerxes3rd]

Commit: https://github.com/Xerxes3rd/Wifiibo/commit/e94aac239d296546c97b67f0e35463517f820b7f

Multiple file upload feature added. Needs some testing, and the amiibo data doesn't show up for each upload (the amiibo data comes over the WebSocket, which gets overloaded during upload). If an upload fails (e.g. duplicate filename) you might miss the message, although it will show up in the browser's console log. Next step is to change the "Upload" popup so it's a message log instead of an amiibo display.

[Xerxes3rd]

After only 9 months, version 1.40 is out, which allows for multi-file uploads, so you can pick multiple files in the file selection dialog. The "Invalid Key File Error" is fixed as well.

If possible, please try this version. If I don't hear anything for a couple weeks, I'll close this issue and it can be re-opened at a later time if necessary.

[Xerxes3rd]

Closing this issue- re-open if necessary.