Reconsider using commercial services for ML tasks like classification

[jeffbl]

Early in the project, we expressed a preference for using open source and freely available ML resources, since this would reduce costs. Now that we've spent time creating our own models, and are getting a better understanding of their capabilities, performance, and costs (e.g. GPU RAM), it seems like a good time to also take another look at commercial services that could expedite progress and/or improve our results.

For example, would Azure services give us a piece of what we need? https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-categorizing-images

Once you've had a chance to think about this, please ping @jeffbl to discuss.

Note that even if we don't end up using any commercial services on the McGill reference IMAGE server, this may be an option for a paid service run by an external company.

[gp1702]

This tool is really cool. We could actually use it as a first level classifier instead of training our own model.

[jeffbl]

Hat tip @Sabrina-Knappe, who mentioned it in a meeting.

[gp1702]

I guess we just need to figure out how much is this going to cost us.

[jeffbl]

Based on our Monday meeting, this is high priority, for accelerating the within-photograph portion of the ML preprocessor chain. @gp1702 can we get a take on this for this upcoming Monday's meeting (Aug 30)?

[jeffbl]

From conversation, key things to determine are:

1. What services are available
2. Capabilities and likely quality
3. Developer issues/estimates for integrating into architecture as preprocessors
4. Rough cost estimate

[gp1702]

One issue I have is that it might not be possible to do a rigorous quality test for many images since it might actually exhaust the free queries budget. I could do a simple test on a few sampled images. Would it be better to test the latency of these services by running them in conjunction with all the other models (especially the high level classifier ) ?

[jeffbl]

Since the preprocessors are serialized right now, probably makes sense to isolate the performance of each preprocessor so we can see what is causing delays. That said, I'd be very surprised if a commercial service would be slower than us running something ourselves.

[jeffbl]

Google's version as well. Note that this, and Microsoft's have object detection with bounding boxes. Roughly $0.0025/query (first 1,000 transactions/month for free), which seems pretty cheap. I don't know if it requires running an instance on their cloud or anything.

https://cloud.google.com/vision/pricing

[gp1702]

Here's the pricing details for the azure services: https://azure.microsoft.com/en-us/pricing/details/cognitive-services/computer-vision/#pricing

[jeffbl]

I'd seen that, and it seemed like that would mean a fraction of a penny per query, which unless we need to make a bunch of queries for each graphic, would be pretty cheap. ~ $0.00128 / query. So even 100k queries / month would end up with something like $128/mo in costs. Not completely trivial, but I think we'd be a raging success if we hit that usage level? It was not clear, to me though, if you also need to run one of their cloud instances in order to access this api? Or is this really the full cost?

[jeffbl]

Calculator for Azure: https://azure.microsoft.com/en-us/pricing/calculator/?service=cognitive-services

[gp1702]

Here's the link to the colab notebook for testing azure services: https://colab.research.google.com/drive/1RYGp5k1276kZAJP_fSy7jGUy0ZbX21wR?usp=sharing. Feel free to test on images you think might be challenging. I think we might need to create new work item for integrating this service in the pipeline.

Note that the current api-key uses my personal credit card (Microsoft will start charging me after the first 1000 queries)

[gp1702]

I think we've zeroed in on using Azure for this. Should this be closed?

[jeffbl]

OK, but first open two new issues, I think:

1. Integrate Azure classifier preprocessor
2. Investigate commercial object detection tools. I think Azure has this as well, but this seems like a separate thing from the classifier. If this would mean reduced GPU use, faster results, or most importantly, better results, than could be another use of commercial services. Thoughts?

[gp1702]

Do we need a new object detector. Afaik we already have one implemented.

[gp1702]

I've created a new issue so closing this for now.