search

projectdiscovery / cloudlist

Cloudlist is a tool for listing Assets from multiple Cloud Providers.
https://projectdiscovery.io
MIT License
816 stars 105 forks source link

[feature] Suggestions for further resources to export #5

Open ResistanceIsUseless opened 3 years ago

ResistanceIsUseless commented 3 years ago

Speaking from personal experience, besides standard VM/NIC assigned IP's it would be so amazing if this could pull all the other endpoints that are publicly available and allow for misconfiguration. These are the examples that come to mind.

AWS:

Azure:

0xtavian commented 2 years ago

Whenever the work on this starts, this site might be helpful to identify similar services offered on different cloud providers https://comparecloud.in/

sullo commented 2 years ago

This should extend to all cloud providers to compile the best dataset possible.

mzpqnxow commented 1 year ago

Not sure if you prefer this as a new issue, or as a comment here or in #2 but I have a few more sources that should be considered if resources are invested into expanding data sources

For some background, I've been working for a few years on a project with similar objectives- exporting "all the things" from a large collection of Cloud providers, CDNs, DNS providers, commercial DNS "OSINT" services, on-prem commercial products, and proprietary/internal sources (e.g. proprietary IPAM/DNS solutions)

The focus is on domains/FQDNs and IPv4/IPv6 addresses and networks

Unfortunately, that work is not my intellectual property as it is developed and maintained for my employer.

That said, I'd be happy to contribute whatever I can get approved for

What I wrote is in Python and has no "special sauce," so getting approval to share information from that work with this project shouldn't be a major challenge

There is significant overlap in the sources supported, but there are a few that are unique to what I have that I may be able to provide an implementation for (or at least a suggestion for implementing the source, brief notes on which endpoints are relevant for which data points, and/or testing)

The sources on my side, some of which would definitely be of interest to this project, others possibly not:

Another thing that may be of some value to the project (and relevant to this specific issue) is a curated set of metadata for the "big three" cloud platforms, focusing on the description of all "resource types" that are useful for asset inventory purposes ("resource type" is the Azure term, I don't recall off the top of my head what AWS and GCP call them)

For this, I maintain 3 large YaML files that have details about GCP, Azure and AWS resources respectively

Each file contains metadata that includes each of the resource type names supported by that platform (e.g. "microsoft.web.sites" is one of several hundred Azure resource types) as well as metadata describing (with jq syntax) how to programmatically extract FQDNs or IP addresses from the structured data returned by the respective API

I can provide that metadata and you can evaluate each resource type to determine if it's suitable for this project. You may even be able to use the extraction rules as a reference, though that depends on which APIs you're using to pull the data

Please let me know if you're interested in the YaML metadata and I can see if I'll be able to contribute it in part or in full (this metadata was developed as part of the larger solution I mentioned, which was not developed under an open-source license)

I can't commit any development time for the other sources I mentioned at this time, though I'd like to if I can find the time in the next 6 months

Thanks for your work on this project. It's always nice (and often very helpful) to see others working on similar problems. It seems this project does this very well

EDIT: Amazingly, I wrote this on mobile, so I apologize for any redundant text or typos; I already notice i mixed up the issue numbers, fixed that