When viewing applications most show as unusable IDs

[wwsean08]

When viewing the spend report for AWS Marketplace service and grouping by product most products are some sort of hash or ID. It would be great if they could be resolved to an actual name, I've had to google these IDs to determine what they are using http://thecloudmarket.com/ to determine what they are and help determine if they are necessary. Below is an example screenshot of the table.

[jessesuen]

Interesting. In fact, Claudia does attempt to look up products by their product code, and display the human readable name. It uses the DescribeImages API to do this. If it is unable to find the name it falls back to the code. It appears it was partially successful with some products (e.g. CodeBuild), but not others. I will have to look at a few these codes to see why it is unable to do this.

[wwsean08]

Yeah the first few ones which I looked up were images no longer available on the marketplace which may have something to do with it. They are old versions of F5 BIG-IP AMI.

[jessesuen]

@wwsean08 - can you check the display names of the marketplace products in Claudia again, and let me know if they eventually got updated? I tried a random sampling of the product codes you listed in this bug, and all the ones I sampled were able to be resolved via the Amazon describe instances query. This means Claudia technically should be able to show the correct display name.

In [35]: ec2.describe_images(Filters=[{'Name': 'product-code','Values': ['cy1odl9nu3xx3n9vuasz5eel0',]}, {'Name': 'owner-id','Values': ['679593333241',]}])['Images'][0]['Description']
Out[35]: 'F5 Networks Hourly BIGIP-13.0.0.2.0.1671 - Best 1Gbps - built on May 01 2017  9_26_59AM - size_129GB'

In [36]: ec2.describe_images(Filters=[{'Name': 'product-code','Values': ['885sdma9j6uwa95dqbw183ulo',]}, {'Name': 'owner-id','Values': ['679593333241',]}])['Images'][0]['Description']
Out[36]: 'F5 Networks Hourly BIGIP-12.1.2.1.0.271 - Best 5Gbps - built on Apr 29 2017 10_55PM - size_126GB'

In [37]: ec2.describe_images(Filters=[{'Name': 'product-code','Values': ['77x8m24ws2lqalxmdr3v9ymow',]}, {'Name': 'owner-id','Values': ['679593333241',]}])['Images'][0]['Description']
Out[37]: 'F5 Networks Hourly BIGIP-11.5.4.2.0.291 - Best 200Mbps - built on Jul 27 2016  9_24PM - size_80GB'

In [38]: ec2.describe_images(Filters=[{'Name': 'product-code','Values': ['6llw0nuwm7pq88qqoo04q7okg',]}, {'Name': 'owner-id','Values': ['679593333241',]}])['Images'][0]['Description']
Out[38]: 'F5 Networks Hourly BIGIP-13.0.0.2.0.1671 - Better 25Mbps - built on May 01 2017  8_00_35AM - size_129GB'

If the product codes are now being resolved to human readable names, it simply means Claudia did not get around to processing it at the time that you viewed the report.

If they are still only showing product codes, and claudia is reporting a status of "current", then something else is preventing the proper lookup.

[wwsean08]

They are still showing the same codes and status is current. I just verified the outbound rule of the security group, it's allowing all traffic everywhere, and the NACL hasn't been changed so that's not blocking it.

[jessesuen]

@wwsean08 - could you ssh to the claudia instance and run the following command and let me know what comes out?

$ docker logs ec2user_ingestd_1  2>&1 | grep cy1odl9nu3xx3n9vuasz5eel0 | tail -20

You can replace cy1odl9nu3xx3n9vuasz5eel0 with any product code that is failed to be identified. The output should indicate the API error of why Claudia can't identify a product.

[wwsean08]

Sure, I'll get back to you in a few days on it though as I'm going to be out on PTO tomorrow and possibly Wednesday.

[wwsean08]

Looks like I probably need to either apply a role to the EC2 instance or do the keys that are used to access the S3 bucket containing the billing information need more than permissions for S3 (not sure if it has that currently given that I didn't create it or have access to view it's permissions).

Which route do I need to go?

2017/09/30 16:24:52 Failed to identify product cy1odl9nu3xx3n9vuasz5eel0: [ERR_INTERNAL] UnauthorizedOperation: You are not authorized to perform this operation.

[jessesuen]

Ah, yes! The policy attached to the access keys should have the DescribeInstance permission as described here:

https://applatix.github.io/claudia/setup/aws/#step-3-create-policy-for-claudia

Can you try that? It may still take a day to see the results, because I believe Claudia will only attempt to re-identify products when there are new reports.

[wwsean08]

Ok I'll reach out to the people that provisioned the IAM user and see what I can do and get back to you.

[wwsean08]

I'm going to close this as it seems pretty obvious this is a problem on our end of not reading part of the documentation or maybe referring to an older version. Thanks for all the help @jessesuen

[jessesuen]

Thanks @wwsean08. Although I believe there is another bug lurking here in that CodeBuild and ElasticMapReduce are showing up in the marketplace category, when they are supposed to be AWS products. I'll file another bug for this

[wwsean08]

So I'm reopening because we fixed the user permissions and checking the logs it should be working:

2017/10/30 17:50:22 Identifying product cy1odl9nu3xx3n9vuasz5eel0 2017/10/30 17:50:23 Product cy1odl9nu3xx3n9vuasz5eel0 identified: ProductCodeId: "cy1odl9nu3xx3n9vuasz5eel0", ProductCodeId: "cy1odl9nu3xx3n9vuasz5eel0", ProductCodeId: "cy1odl9nu3xx3n9vuasz5eel0", ProductCodeId: "cy1odl9nu3xx3n9vuasz5eel0", 2017/10/30 17:50:23 Upserted product {cy1odl9nu3xx3n9vuasz5eel0 F5 Networks Hourly BIGIP-12.1.2.1.0.271 - Best 1Gbps - built on Apr 29 2017 -daab4895-5354-47ee-b8cd-6833658d1254-ami-70d0b166.4 F5 Networks Hourly BIGIP-12.1.2.1.0.271 - Best 1Gbps - built on Apr 29 2017 10_49PM - size_126GB}

But unfortunately when I go and view the data it still shows that product id, even when i just limit it to today (we've had the permissions changed since friday). I'm thinking I should just respin this up, do you think that'll fix it?

[jessesuen]

Hi @wwsean08, sorry I missed this message. If you see those logs, I actually expect the UI to display the correct information. The only thing I can think of, is if the SQL transaction got aborted due to some error occurring later. I do think respinning the app would fix it, but then again, I expected it to work after seeing those logs. Did you try relaunching the app?

[wwsean08]

I think i restarted the whole instance, but i'll do that just to be sure.

[wwsean08]

Restarting the application didn't fix it but creating a new one did. Strange but a problem for another day. In creating the new one I also upgraded from 1.09 to 1.10 so maybe there was a 1.09 bug that was causing it. Feel free to close if you want.