search

opensearch-project / opensearch-build

🧰 OpenSearch / OpenSearch-Dashboards Build Systems
Apache License 2.0
140 stars 273 forks source link

[Bug]: OpenSearch Dashboards needs to rebuild core every time (make --component work) #606

Open dblock opened 3 years ago

dblock commented 3 years ago

Describe the bug

OpenSearch dashboards requires to rebuild core every single time to build any of the components, which makes it very slow to debug component build problems.

To reproduce

./build-workflow/build.sh manifests/1.1.0/openseach-dashboards-1.1.0.yml

Succeeds for all components.

./build-workflow/build.sh manifests/1.1.0/openseach-dashboards-1.1.0.yml --component=alertingDashboards

Fails with

+ PLUGIN_NAME=alertingDashboards
+ cp -r ../alertingDashboards/ ../OpenSearch-Dashboards/plugins
cp: cannot create directory '../OpenSearch-Dashboards/plugins': No such file or directory
2021-09-27 14:17:21 ERROR    Error building alertingDashboards, retry with: ./bundle-workflow/build.sh manifests/1.1.0/opensearch-dashboards-1.1.0.yml --component alertingDashboards
Traceback (most recent call last):
  File "./bundle-workflow/src/run_build.py", line 80, in <module>
    sys.exit(main())
  File "./bundle-workflow/src/run_build.py", line 66, in main
    builder.build(target)
  File "/home/ubuntu/source/opensearch-project/opensearch-build/dblock-opensearch-build/bundle-workflow/src/build_workflow/builder.py", line 41, in build
    self.git_repo.execute(build_command)
  File "/home/ubuntu/source/opensearch-project/opensearch-build/dblock-opensearch-build/bundle-workflow/src/git/git_repository.py", line 70, in execute
    subprocess.check_call(command, cwd=cwd, shell=True)
  File "/home/ubuntu/.pyenv/versions/3.7.11/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/home/ubuntu/source/opensearch-project/opensearch-build/dblock-opensearch-build/bundle-workflow/scripts/default/build.sh -v 1.1.0 -a x64 -s false -o artifacts -d true' returned non-zero exit status 1.

Expected behavior

We could adopt something similar to how OpenSearch builds into ~/.m2 for dependencies with npm local, so that the output from the dashboards core could be published and preserved.

kavilla commented 3 years ago

Subtask: https://github.com/opensearch-project/opensearch-build/issues/680

Tengda-He commented 3 years ago

Had a offline sync with kavilla, thinking the right solution should be letting plugin owner figure out what modules are required from OSD for plugin build. The current situation is we have many plugins, and each has a different dependency hierarchy with OSD modules. (And some plugin have import module with relative path pointing to OSD src folder). This makes it quite hard to develop a fit-all solution from OSD side. Prefer to hand this work to plugin owner to achieve.

kavilla commented 3 years ago

We were thinking about to solve this issue we will allow for multiple components to be passed so it builds together

peterzhuamazon commented 2 years ago

1278 provides a way to get it run.

We can try to use this for the fixes.

gaiksaya commented 2 years ago

One way might be to use local-npm (similar to maven local). Build dashboards and push to local-npm (one time) and then build the components on top of it. I have not tested this myself but can be one of the option.

bbarani commented 2 years ago

@kavilla @tianleh Is this still an open issue?

peterzhuamazon commented 2 years ago

Since we already implement the --component on both build and jenkins I think this can be closed. Thanks.

dblock commented 2 years ago

This problem is about developers, not Jenkins, see description of the issue. I don't think it should be closed.

bbarani commented 1 year ago

@seanneumann @kavilla This needs to be prioritized to improve the build generation velocity. Can you please prioritize this as soon as possible?

seanneumann commented 1 year ago

Hey @bbarani. Taking a look with the team now.

kavilla commented 1 year ago

@seanneumann @kavilla This needs to be prioritized to improve the build generation velocity. Can you please prioritize this as soon as possible?

Right now it does re-use the build from OpenSearch Dashboards in the default script:

I don't think this will improve build generation velocity besides if we only just want to build a singular component which based on Peter's comment it is supported on Jenkins. Solving this would allow for developers locally to push out if they wanted to build out --component securityDashboards.

Is there a case that this impacting @bbarani?

bbarani commented 1 year ago

@kavilla As far as I know, this is needed for granular build generation process work that is described here. @peterzhuamazon, you mentioned that we can overcome this blocker by Bootstrapping the core, is that a long term solution?

bbarani commented 1 year ago

The bootstrapping also causes the core build to fail if any of the plugins fail to build as well

ERROR [bootstrap] failed:
ERROR Error: Command failed with exit code 127: /usr/share/opensearch/.nvm/versions/node/v14.20.1/lib/node_modules/yarn/bin/yarn.js install --non-interactive
          at makeError (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:25150:11)
          at handlePromise (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:24085:26)
          at processTicksAndRejections (internal/process/task_queues.js:95:5)
          at async installInDir (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:23821:3)
          at async Project.installDependencies (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:14780:5)
          at async Object.run (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:9003:11)
          at async runCommand (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:57870:5)
          at async Module.run (/tmp/tmpj52kld9y/OpenSearch-Dashboards/packages/osd-pm/dist/index.js:263:3)
error Command failed with exit code 1.
gaiksaya commented 11 months ago

I found a couple of solutions in this article https://medium.com/@debshish.pal/publish-a-npm-package-locally-for-testing-9a00015eb9fd which can be used to publish opensearch-dashboards as a local npm package rather than bootstraping everytime (if I understand if correctly). Please take a look and see if this can be helpful. Also not sure if @Tengda-He is still working on this issue. If not please assign a new assignee. Removing the assignee for now.

Thanks!

gaiksaya commented 2 months ago

@kavilla @AMoo-Miki @ashwin-pc Wondering if you have any inputs for this issue? Thanks!

AMoo-Miki commented 4 weeks ago

I am not sure I understand the problem enough or the historical context of this issue.