search

awslabs / aws-service-catalog-factory

This is a framework where you define a Service Catalog portfolio, products and versions using YAML. For versions of your products you specify where the source code for them can be found and the framework publishes the portfolio, products and versions in every* AWS Region after validating, linting and testing them.
Apache License 2.0
136 stars 44 forks source link

Service Catalog Factory pipeline fails building example porfolio. Fails at GenerateConstraints (v0.83) #276

Closed horrockp closed 2 years ago

horrockp commented 2 years ago

Please ensure you are using the latest version and have tried to run a validate command on your source files

Steps to reproduce

  1. follwed this guide https://service-catalog-tools-workshop.com/2021-workshop/300-provisioning-subnets/15-using-service-catalog.html

Actual results

INFO MainThread Finished building up portfolio list: {'networking-mandatory': []}

File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/servicecatalog_factory/commands/portfolios.py", line 752, in generate_launch_constraints

FileNotFoundError: [Errno 2] No such file or directory: 'output/CreatePortfolioTask/eu-central-1-networking-mandatory.json

[Container] 2022/08/09 11:18:15 Command did not exit successfully servicecatalog-factory --info generate-launch-constraints portfolios/ exit status 1

Ive noticed that the file name in the output/CreatePortfolioTask has an appended "-" on the end

-rw-r--r-- 1 root root 337 Aug 9 13:02 eu-central-1-networking-mandatory-.json

sandrich commented 2 years ago

same issue

shydhanya commented 2 years ago

same issue.

sandrich commented 2 years ago

Seems that this breaks the change

https://github.com/awslabs/aws-service-catalog-factory/commit/1319e9a27ac02d7a22340dd737baedf7b7389a67#diff-9f897edfd24c513f05dc0fec71146fc69d91a0d03d514078245a92091e3c7ed7R25

eamonnfaherty commented 2 years ago

can you share a full error message please and ideally the yaml file you have written

sandrich commented 2 years ago

Here the full output

[Container] 2022/08/23 19:53:00 Waiting for agent ping
--
2 | [Container] 2022/08/23 19:53:01 Waiting for DOWNLOAD_SOURCE
3 | [Container] 2022/08/23 19:53:02 Phase is DOWNLOAD_SOURCE
4 | [Container] 2022/08/23 19:53:02 CODEBUILD_SRC_DIR=/codebuild/output/src153902188/src
5 | [Container] 2022/08/23 19:53:02 YAML location is /codebuild/readonly/buildspec.yml
6 | [Container] 2022/08/23 19:53:02 Found possible syntax errors in buildspec:
7 | The following keys cannot be identified:
8 | Tags, TimeoutInMinutes
9 | [Container] 2022/08/23 19:53:02 Processing environment variables
10 | [Container] 2022/08/23 19:53:02 Selecting 'python' runtime version '3.7' based on manual selections...
11 | [Container] 2022/08/23 19:53:02 Running command echo "Installing Python version 3.7 ..."
12 | Installing Python version 3.7 ...
13 |  
14 | [Container] 2022/08/23 19:53:02 Running command pyenv global  $PYTHON_37_VERSION
15 |  
16 | [Container] 2022/08/23 19:53:05 Moving to directory /codebuild/output/src153902188/src
17 | [Container] 2022/08/23 19:53:05 Configuring ssm agent with target id: codebuild:08ac4b89-6403-4f31-8c3e-2dad6fe54ad3
18 | [Container] 2022/08/23 19:53:05 Successfully updated ssm agent configuration
19 | [Container] 2022/08/23 19:53:05 Registering with agent
20 | [Container] 2022/08/23 19:53:05 Phases found in YAML: 2
21 | [Container] 2022/08/23 19:53:05  INSTALL: 1 commands
22 | [Container] 2022/08/23 19:53:05  BUILD: 2 commands
23 | [Container] 2022/08/23 19:53:05 Phase complete: DOWNLOAD_SOURCE State: SUCCEEDED
24 | [Container] 2022/08/23 19:53:05 Phase context status code:  Message:
25 | [Container] 2022/08/23 19:53:05 Entering phase INSTALL
26 | [Container] 2022/08/23 19:53:05 Running command pip install aws-service-catalog-factory==0.84.0
27 | Collecting aws-service-catalog-factory==0.84.0
28 | Downloading aws-service-catalog-factory-0.84.0.tar.gz (76 kB)
29 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.8/76.8 kB 2.6 MB/s eta 0:00:00
30 | Installing build dependencies: started
31 | Installing build dependencies: finished with status 'done'
32 | Getting requirements to build wheel: started
33 | Getting requirements to build wheel: finished with status 'done'
34 | Preparing metadata (pyproject.toml): started
35 | Preparing metadata (pyproject.toml): finished with status 'done'
36 | Collecting s3transfer==0.5.0
37 | Downloading s3transfer-0.5.0-py3-none-any.whl (79 kB)
38 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.2/79.2 kB 6.0 MB/s eta 0:00:00
39 | Collecting tornado==4.5.3
40 | Downloading tornado-4.5.3.tar.gz (484 kB)
41 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 484.2/484.2 kB 34.8 MB/s eta 0:00:00
42 | Preparing metadata (setup.py): started
43 | Preparing metadata (setup.py): finished with status 'done'
44 | Collecting terminaltables==3.1.0
45 | Downloading terminaltables-3.1.0.tar.gz (12 kB)
46 | Preparing metadata (setup.py): started
47 | Preparing metadata (setup.py): finished with status 'done'
48 | Collecting urllib3==1.26.7
49 | Downloading urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
50 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 138.8/138.8 kB 10.6 MB/s eta 0:00:00
51 | Collecting enum34==1.1.10
52 | Downloading enum34-1.1.10-py3-none-any.whl (11 kB)
53 | Collecting typing-extensions==3.10.0.2
54 | Downloading typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)
55 | Collecting better-boto==0.43.0
56 | Downloading better_boto-0.43.0-py3-none-any.whl (13 kB)
57 | Collecting PyYAML==5.4
58 | Downloading PyYAML-5.4-cp37-cp37m-manylinux1_x86_64.whl (636 kB)
59 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 636.5/636.5 kB 31.4 MB/s eta 0:00:00
60 | Collecting troposphere==3.1.0
61 | Downloading troposphere-3.1.0-py3-none-any.whl (242 kB)
62 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.6/242.6 kB 14.1 MB/s eta 0:00:00
63 | Collecting click==7.0
64 | Downloading Click-7.0-py2.py3-none-any.whl (81 kB)
65 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 81.3/81.3 kB 5.6 MB/s eta 0:00:00
66 | Collecting requests==2.26.0
67 | Downloading requests-2.26.0-py2.py3-none-any.whl (62 kB)
68 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.3/62.3 kB 4.4 MB/s eta 0:00:00
69 | Collecting idna==2.8
70 | Downloading idna-2.8-py2.py3-none-any.whl (58 kB)
71 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.6/58.6 kB 3.4 MB/s eta 0:00:00
72 | Collecting colorclass==2.2.0
73 | Downloading colorclass-2.2.0.tar.gz (17 kB)
74 | Preparing metadata (setup.py): started
75 | Preparing metadata (setup.py): finished with status 'done'
76 | Collecting python-daemon==2.1.2
77 | Downloading python_daemon-2.1.2-py2.py3-none-any.whl (19 kB)
78 | Collecting lockfile==0.12.2
79 | Downloading lockfile-0.12.2-py2.py3-none-any.whl (13 kB)
80 | Requirement already satisfied: MarkupSafe==2.0.1 in /root/.pyenv/versions/3.7.13/lib/python3.7/site-packages (from aws-service-catalog-factory==0.84.0) (2.0.1)
81 | Requirement already satisfied: six==1.16.0 in /root/.pyenv/versions/3.7.13/lib/python3.7/site-packages (from aws-service-catalog-factory==0.84.0) (1.16.0)
82 | Collecting luigi==2.8.6
83 | Downloading luigi-2.8.6.tar.gz (1.2 MB)
84 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 55.1 MB/s eta 0:00:00
85 | Preparing metadata (setup.py): started
86 | Preparing metadata (setup.py): finished with status 'done'
87 | Collecting yamale==3.0.8
88 | Downloading yamale-3.0.8-py3-none-any.whl (51 kB)
89 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.5/51.5 kB 1.5 MB/s eta 0:00:00
90 | Collecting charset-normalizer==2.0.7
91 | Downloading charset_normalizer-2.0.7-py3-none-any.whl (38 kB)
92 | Collecting cfn-flip==1.2.3
93 | Downloading cfn_flip-1.2.3.tar.gz (10 kB)
94 | Preparing metadata (setup.py): started
95 | Preparing metadata (setup.py): finished with status 'done'
96 | Collecting deepmerge==0.3.0
97 | Downloading deepmerge-0.3.0-py2.py3-none-any.whl (7.6 kB)
98 | Requirement already satisfied: Jinja2==2.11.3 in /root/.pyenv/versions/3.7.13/lib/python3.7/site-packages (from aws-service-catalog-factory==0.84.0) (2.11.3)
99 | Collecting boto3==1.19.2
100 | Downloading boto3-1.19.2-py3-none-any.whl (131 kB)
101 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 131.6/131.6 kB 32.3 MB/s eta 0:00:00
102 | Requirement already satisfied: python-dateutil==2.8.2 in /root/.pyenv/versions/3.7.13/lib/python3.7/site-packages (from aws-service-catalog-factory==0.84.0) (2.8.2)
103 | Collecting botocore==1.22.2
104 | Downloading botocore-1.22.2-py3-none-any.whl (8.0 MB)
105 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 87.2 MB/s eta 0:00:00
106 | Collecting docutils==0.14
107 | Downloading docutils-0.14-py3-none-any.whl (543 kB)
108 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 543.2/543.2 kB 18.6 MB/s eta 0:00:00
109 | Collecting certifi==2021.10.8
110 | Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
111 | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 149.2/149.2 kB 4.6 MB/s eta 0:00:00
112 | Requirement already satisfied: jmespath==0.10.0 in /root/.pyenv/versions/3.7.13/lib/python3.7/site-packages (from aws-service-catalog-factory==0.84.0) (0.10.0)
113 | Requirement already satisfied: setuptools in /root/.pyenv/versions/3.7.13/lib/python3.7/site-packages (from python-daemon==2.1.2->aws-service-catalog-factory==0.84.0) (57.4.0)
114 | Building wheels for collected packages: aws-service-catalog-factory, cfn-flip, colorclass, luigi, terminaltables, tornado
115 | Building wheel for aws-service-catalog-factory (pyproject.toml): started
116 | Building wheel for aws-service-catalog-factory (pyproject.toml): finished with status 'done'
117 | Created wheel for aws-service-catalog-factory: filename=aws_service_catalog_factory-0.84.0-py3-none-any.whl size=136617 sha256=d14431ecc28f7b90db8d60f010c925c305d247650331729ae25245b2e5b96bb7
118 | Stored in directory: /root/.cache/pip/wheels/b5/cf/f1/0cd8c1c3af5b47b14131a2c8177572d686f7e8c8f254251f34
119 | Building wheel for cfn-flip (setup.py): started
120 | Building wheel for cfn-flip (setup.py): finished with status 'done'
121 | Created wheel for cfn-flip: filename=cfn_flip-1.2.3-py3-none-any.whl size=15150 sha256=4cb20965282fd61cbb8b88d1c0bd617420de46cdf4d2d4e06bb7fe4f2600fe09
122 | Stored in directory: /root/.cache/pip/wheels/32/1f/02/56d8fb5d7f876a2583576eab65c91e85ce18a25db6cb667482
123 | Building wheel for colorclass (setup.py): started
124 | Building wheel for colorclass (setup.py): finished with status 'done'
125 | Created wheel for colorclass: filename=colorclass-2.2.0-py3-none-any.whl size=19394 sha256=65d9d26c3a3684677e62102edd5f7b8d040114c3693e5456f88cb9ebd98ae6fd
126 | Stored in directory: /root/.cache/pip/wheels/fe/96/67/750617a6039bb687b993830c5852beba03d777e2d5b0b06938
127 | Building wheel for luigi (setup.py): started
128 | Building wheel for luigi (setup.py): finished with status 'done'
129 | Created wheel for luigi: filename=luigi-2.8.6-py3-none-any.whl size=1079876 sha256=47d6887742eb9de07c3d794c7a53652109714c8084c16a25f293318dd8ce4569
130 | Stored in directory: /root/.cache/pip/wheels/6f/6d/aa/bbea236376478753761fc5715c7f7eef0bd3c4851cd3b24376
131 | Building wheel for terminaltables (setup.py): started
132 | Building wheel for terminaltables (setup.py): finished with status 'done'
133 | Created wheel for terminaltables: filename=terminaltables-3.1.0-py3-none-any.whl size=15355 sha256=a66e6b83903319e68707759e22f3757248d71c0741dd1014d96083d336e5bd97
134 | Stored in directory: /root/.cache/pip/wheels/ba/ad/c8/2d98360791161cd3db6daf6b5e730f34021fc9367d5879f497
135 | Building wheel for tornado (setup.py): started
136 | Building wheel for tornado (setup.py): finished with status 'done'
137 | Created wheel for tornado: filename=tornado-4.5.3-cp37-cp37m-linux_x86_64.whl size=434073 sha256=e6b0e683bb4a41c549dded959a5ccfa3c4ecd2c9e7bb402a5966814d1c5c769f
138 | Stored in directory: /root/.cache/pip/wheels/a2/45/43/36ec7a893e16c1212a6b1505ded0a2d73cf8e863a0227c8e04
139 | Successfully built aws-service-catalog-factory cfn-flip colorclass luigi terminaltables tornado
140 | Installing collected packages: typing-extensions, tornado, terminaltables, lockfile, enum34, docutils, colorclass, certifi, urllib3, PyYAML, python-daemon, idna, deepmerge, click, charset-normalizer, yamale, requests, luigi, cfn-flip, botocore, troposphere, s3transfer, boto3, better-boto, aws-service-catalog-factory
141 | Attempting uninstall: typing-extensions
142 | Found existing installation: typing-extensions 3.10.0.0
143 | Uninstalling typing-extensions-3.10.0.0:
144 | Successfully uninstalled typing-extensions-3.10.0.0
145 | Attempting uninstall: docutils
146 | Found existing installation: docutils 0.16
147 | Uninstalling docutils-0.16:
148 | Successfully uninstalled docutils-0.16
149 | Attempting uninstall: certifi
150 | Found existing installation: certifi 2022.6.15
151 | Uninstalling certifi-2022.6.15:
152 | Successfully uninstalled certifi-2022.6.15
153 | Attempting uninstall: urllib3
154 | Found existing installation: urllib3 1.26.9
155 | Uninstalling urllib3-1.26.9:
156 | Successfully uninstalled urllib3-1.26.9
157 | Attempting uninstall: PyYAML
158 | Found existing installation: PyYAML 5.4.1
159 | Uninstalling PyYAML-5.4.1:
160 | Successfully uninstalled PyYAML-5.4.1
161 | Attempting uninstall: idna
162 | Found existing installation: idna 2.10
163 | Uninstalling idna-2.10:
164 | Successfully uninstalled idna-2.10
165 | Attempting uninstall: click
166 | Found existing installation: click 7.1.2
167 | Uninstalling click-7.1.2:
168 | Successfully uninstalled click-7.1.2
169 | Attempting uninstall: requests
170 | Found existing installation: requests 2.25.1
171 | Uninstalling requests-2.25.1:
172 | Successfully uninstalled requests-2.25.1
173 | Attempting uninstall: botocore
174 | Found existing installation: botocore 1.27.18
175 | Uninstalling botocore-1.27.18:
176 | Successfully uninstalled botocore-1.27.18
177 | Attempting uninstall: s3transfer
178 | Found existing installation: s3transfer 0.6.0
179 | Uninstalling s3transfer-0.6.0:
180 | Successfully uninstalled s3transfer-0.6.0
181 | Attempting uninstall: boto3
182 | Found existing installation: boto3 1.24.18
183 | Uninstalling boto3-1.24.18:
184 | Successfully uninstalled boto3-1.24.18
185 | ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
186 | awscli 1.25.18 requires botocore==1.27.18, but you have botocore 1.22.2 which is incompatible.
187 | awscli 1.25.18 requires s3transfer<0.7.0,>=0.6.0, but you have s3transfer 0.5.0 which is incompatible.
188 | aws-sam-translator 1.46.0 requires boto3==1.*,>=1.19.5, but you have boto3 1.19.2 which is incompatible.
189 | aws-sam-cli 1.52.0 requires boto3==1.*,>=1.19.5, but you have boto3 1.19.2 which is incompatible.
190 | aws-sam-cli 1.52.0 requires click~=7.1, but you have click 7.0 which is incompatible.
191 | aws-sam-cli 1.52.0 requires requests==2.25.1, but you have requests 2.26.0 which is incompatible.
192 | aws-sam-cli 1.52.0 requires typing-extensions==3.10.0.0, but you have typing-extensions 3.10.0.2 which is incompatible.
193 | Successfully installed PyYAML-5.4 aws-service-catalog-factory-0.84.0 better-boto-0.43.0 boto3-1.19.2 botocore-1.22.2 certifi-2021.10.8 cfn-flip-1.2.3 charset-normalizer-2.0.7 click-7.0 colorclass-2.2.0 deepmerge-0.3.0 docutils-0.14 enum34-1.1.10 idna-2.8 lockfile-0.12.2 luigi-2.8.6 python-daemon-2.1.2 requests-2.26.0 s3transfer-0.5.0 terminaltables-3.1.0 tornado-4.5.3 troposphere-3.1.0 typing-extensions-3.10.0.2 urllib3-1.26.7 yamale-3.0.8
194 | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
195 |  
196 | [notice] A new release of pip available: 22.1.2 -> 22.2.2
197 | [notice] To update, run: pip install --upgrade pip
198 |  
199 | [Container] 2022/08/23 19:53:54 Phase complete: INSTALL State: SUCCEEDED
200 | [Container] 2022/08/23 19:53:54 Phase context status code:  Message:
201 | [Container] 2022/08/23 19:53:54 Entering phase PRE_BUILD
202 | [Container] 2022/08/23 19:53:54 Phase complete: PRE_BUILD State: SUCCEEDED
203 | [Container] 2022/08/23 19:53:54 Phase context status code:  Message:
204 | [Container] 2022/08/23 19:53:54 Entering phase BUILD
205 | [Container] 2022/08/23 19:53:54 Running command servicecatalog-factory --info generate-launch-constraints portfolios/
206 | INFO MainThread generating launch contraints
207 | INFO MainThread Building up portfolios list
208 | INFO MainThread Loading portfolio: portfolios//account-vending.yaml
209 | INFO MainThread Checking for external config
210 | INFO MainThread Loading portfolio: portfolios//config-rules.yaml
211 | INFO MainThread Checking for external config
212 | INFO MainThread Finished building up portfolio list: {'account-vending-saas-lz-account-vending': [], 'config-rules-saas-lz-security-tools': []}
213 | INFO MainThread starting to write the template
214 | INFO MainThread looking at region eu-west-1
215 | Traceback (most recent call last):
216 | File "/root/.pyenv/versions/3.7.13/bin/servicecatalog-factory", line 8, in <module>
217 | sys.exit(cli())
218 | File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/click/core.py", line 764, in __call__
219 | return self.main(*args, **kwargs)
220 | File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/click/core.py", line 717, in main
221 | rv = self.invoke(ctx)
222 | File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
223 | return _process_result(sub_ctx.command.invoke(sub_ctx))
224 | File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/click/core.py", line 956, in invoke
225 | return ctx.invoke(self.callback, **ctx.params)
226 | File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/click/core.py", line 555, in invoke
227 | return callback(*args, **kwargs)
228 | File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/servicecatalog_factory/cli.py", line 621, in generate_launch_constraints
229 | portfolios.generate_launch_constraints(p)
230 | File "/root/.pyenv/versions/3.7.13/lib/python3.7/site-packages/servicecatalog_factory/commands/portfolios.py", line 752, in generate_launch_constraints
231 | "r",
232 | FileNotFoundError: [Errno 2] No such file or directory: 'output/CreatePortfolioTask/eu-west-1-account-vending-saas-lz-account-vending.json'
233 |  
234 | [Container] 2022/08/23 19:53:55 Command did not exit successfully servicecatalog-factory --info generate-launch-constraints portfolios/ exit status 1
235 | [Container] 2022/08/23 19:53:55 Phase complete: BUILD State: FAILED
236 | [Container] 2022/08/23 19:53:55 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: servicecatalog-factory --info generate-launch-constraints portfolios/. Reason: exit status 1
237 | [Container] 2022/08/23 19:53:55 Entering phase POST_BUILD
238 | [Container] 2022/08/23 19:53:55 Phase complete: POST_BUILD State: SUCCEEDED
239 | [Container] 2022/08/23 19:53:55 Phase context status code:  Message:
240 | [Container] 2022/08/23 19:53:55 Expanding base directory path: .
241 | [Container] 2022/08/23 19:53:55 Assembling file list
242 | [Container] 2022/08/23 19:53:55 Expanding .
243 | [Container] 2022/08/23 19:53:55 Expanding file paths for base directory .
244 | [Container] 2022/08/23 19:53:55 Assembling file list
245 | [Container] 2022/08/23 19:53:55 Expanding results/*/*
246 | [Container] 2022/08/23 19:53:55 Expanding output/*/*
247 | [Container] 2022/08/23 19:53:55 Expanding output/constraints/launch-role/*.template.yaml
248 | [Container] 2022/08/23 19:53:55 Found 88 file(s)
249 | [Container] 2022/08/23 19:53:55 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
250 | [Container] 2022/08/23 19:53:55 Phase context status code:  Message:

Here the folder content

image
eamonnfaherty commented 2 years ago

0.82.0 introduced a new way to create portfolios (using the products folder). This resulted in the adding of "portfolio_name": self.portfolio_name, to the params_for_results_display of the CreatePortfolioTask which results in a different filename in the output dictory.

eamonnfaherty commented 2 years ago

fixed in https://github.com/awslabs/aws-service-catalog-factory/releases/tag/0.86.0