Mongo'da ne yaptık akış olarak anlatalım

hcipek commented 2 years ago

Apollo EC2 deki datalar, yazılan Python scriptler ile Http Post kullanılarak Apollo Api EC2 üzerinde çalışan Mongo API uygulamasına iletildi. Bu API alınan istekleri işleyerek Apacha Kafka üzerindeki her dile ait olan topiclere yazdı. Bu topiclerdeki data mongo db ye aktarıldı. Ardından Apollo Api EC2 üzerindeki Merger servisi bu iletilen datayı birleştirerek ilgili dosyalara ve commitlere çevirdi. Son olarak oluşan commitler kullanılarak repositoryler toplandı.

hcipek commented 2 years ago

Python scriptler: -İlgili programlama diline ait csv pandas ile okunur. -Row by row giderek her row json a çevrilir. -Bu Jsonlar HTTP Post ile Mongo API a iletilir.

hcipek commented 2 years ago

Mongo API: -/create endpointi üzerinden gelen dataları alır. -Bu datayı dile göre ayrıştırarak ilgili Kafka Topic e yazar. -Kafka topic e düşen datalar MongoDB üzerinde yer alan ilgili collectionlara insert edilir.

hcipek commented 2 years ago

Merger Service: -/api/apollo/github/commits/merge/{language} endpointi üzerinden ilgili dil verilerek tetiklenir. -Bu method verilen dile göre ilgili collectiondaki snippet datasını çeker. -Bu snippetler dosya adı ve snippet dosya sırası kullanılarak sıralanır. -Bu sıra üzerinden aynı dosyaya ait snippetler içerikleri birleştirilir. -Bu birleştirme esnasında snippetin ait olduğu commit datası oluşturulur. Commit oluşturulurken, içerisine mevcut birleştirilen dosya adı eklenir, Github API üzerinden commite ait elimizdeki veride yer almayan bilgiler eklenir.(author name, date etc) -Eğer daha önce başka bir dosya tarafından bu commit oluşturulmuş ise mevcut birleştirilen dosya bu commit içerisindeki dosyalara eklenir. Commit çoklaması olmaz. -Commit oluşturulduktan sonra Commits Collection'a insert edilir. Bu commit'e ait her yeni dosya için update edilerek ilerlenir. -Snippet birleşimi update edilerek gitmek yerine mevcut snippetler dosya adı kullanarak gruplama yapılır. -Her dosya birleşimi bittiğinde CommitFile Collection'a insert edilir. -Bir dile ait tüm snippetler tamamlandığında o dile ait tüm dosyalar birleşmiş halde elimizde yer alır.

-/api/apollo/githun/repos/create endpointi mevcut commitler üzerinden repositoryleri oluşturur. -Elimizdeki dataset içerisinde repository ait url bilgisi bulunmaktadır, bu sebepten bu bilgiyi kullanarak Github API üzerinden repository sorgusu atarak bu repo hakkında aranan bilgiler toplanır. -Bu bilgilerle beraber ilişkili olduğu commit bilgisini ekleyerek oluşan son datayı Repositories Collection'a insert edilir.

hcipek commented 2 years ago

Örnek Repository Data:

_id: ObjectId('6237c8ef1e08d040c90fffbf'),
repoUrl: 'https://github.com/tensorflow/tensorflow',
fullName: 'tensorflow/tensorflow',
ownerLogin: 'tensorflow',
ownerType: 'Organization',
description: 'An Open Source Machine Learning Framework for Everyone',
createdAt: '2015-11-07T01:19:20Z',
updatedAt: '2022-03-20T23:46:48Z',
pushedAt: '2022-03-20T17:59:19Z',
licenseName: 'Apache License 2.0',
licenseUrl: 'https://api.github.com/licenses/apache-2.0',
language: 'C++',
visibilityStatus: 'public',
size: 870901,
stargazersCount: 163715,
watchersCount: 163715,
forksCount: 86510,
openIssuesCount: 2642,
networkCount: 86510,
subscribersCount: 7896,
forkStatus: false,
allowForkingStatus: true,
templateStatus: false,
privateStatus: false,
topics: [
    'deep-learning',
    'deep-neural-networks',
    'distributed',
    'machine-learning',
    'ml',
    'neural-network',
    'python',
    'tensorflow'
],
commits: [
    '686406b968467e7e226fb86601af0f8616ff9736'
],
_class: 'com.apollo.merger.model.github.GithubRepository'

hcipek commented 2 years ago

Örnek Commits Data:

_id: ObjectId('622efd4322d80003aa1117b1'),
language: 'Python',
repoUrl: 'https://github.com/quantopian/zipline',
license: 'Apache-2.0',
commitHash: '014f1fc339dc8b7671d29be2d85ce57d3daec343',
files: [
    'quantopian/zipline/zipline/finance/blotter/blotter.py',
    'quantopian/zipline/setup.py',
    'quantopian/zipline/zipline/data/bcolz_daily_bars.py',
    'quantopian/zipline/zipline/utils/sharedoc.py',
    'quantopian/zipline/zipline/testing/__init__.py',
    'quantopian/zipline/zipline/data/data_portal.py',
    'quantopian/zipline/zipline/finance/asset_restrictions.py',
    'quantopian/zipline/zipline/lib/adjusted_array.py',
    'quantopian/zipline/tests/finance/test_commissions.py',
    'quantopian/zipline/zipline/utils/metautils.py',
    'quantopian/zipline/zipline/finance/blotter/simulation_blotter.py',
    'quantopian/zipline/versioneer.py',
    'quantopian/zipline/zipline/pipeline/data/dataset.py',
    'quantopian/zipline/zipline/pipeline/classifiers/classifier.py',
    'quantopian/zipline/zipline/utils/formatting.py',
    'quantopian/zipline/tests/data/test_resample.py',
    'quantopian/zipline/tests/test_clock.py',
    'quantopian/zipline/zipline/sources/requests_csv.py',
    'quantopian/zipline/zipline/utils/cli.py',
    'quantopian/zipline/zipline/data/fx/utils.py',
    'quantopian/zipline/zipline/examples/olmar.py',
    'quantopian/zipline/zipline/pipeline/filters/filter.py',
    'quantopian/zipline/tests/resources/quandl_samples/rebuild_samples.py',
    'quantopian/zipline/zipline/testing/predicates.py',
    'quantopian/zipline/zipline/data/session_bars.py',
    'quantopian/zipline/zipline/gens/composites.py',
    'quantopian/zipline/tests/test_memoize.py',
    'quantopian/zipline/tests/events/test_events_cme.py',
    'quantopian/zipline/zipline/lib/quantiles.py',
    'quantopian/zipline/zipline/utils/algo_instance.py',
    'quantopian/zipline/tests/pipeline/test_term.py',
    'quantopian/zipline/zipline/lib/normalize.py',
    'quantopian/zipline/tests/test_registration_manager.py',
    'quantopian/zipline/tests/test_labelarray.py',
    'quantopian/zipline/zipline/data/fx/in_memory.py',
    'quantopian/zipline/tests/pipeline/test_classifier.py',
    'quantopian/zipline/zipline/utils/range.py',
    'quantopian/zipline/tests/data/bundles/test_core.py',
    'quantopian/zipline/tests/test_assets.py',
    'quantopian/zipline/zipline/pipeline/factors/events.py',
    'quantopian/zipline/zipline/utils/argcheck.py',
    'quantopian/zipline/tests/test_blotter.py',
    'quantopian/zipline/zipline/__main__.py',
    'quantopian/zipline/zipline/currency.py',
    'quantopian/zipline/tests/metrics/test_metrics.py',
    'quantopian/zipline/ci/make_conda_packages.py',
    'quantopian/zipline/zipline/data/fx/exploding.py',
    'quantopian/zipline/zipline/examples/dual_moving_average.py',
    'quantopian/zipline/zipline/sources/test_source.py',
    'quantopian/zipline/zipline/utils/string_formatting.py',
    'quantopian/zipline/zipline/data/bundles/__init__.py',
    'quantopian/zipline/tests/pipeline/test_blaze.py',
    'quantopian/zipline/tests/test_fetcher.py',
    'quantopian/zipline/tests/test_execution_styles.py',
    'quantopian/zipline/tests/pipeline/test_pipeline_algo.py',
    'quantopian/zipline/zipline/assets/asset_db_migrations.py',
    'quantopian/zipline/zipline/finance/slippage.py',
    'quantopian/zipline/tests/test_history.py',
    'quantopian/zipline/zipline/protocol.py',
    'quantopian/zipline/tests/data/test_hdf5_daily_bars.py',
    'quantopian/zipline/tests/pipeline/test_domain.py',
    'quantopian/zipline/zipline/finance/order.py',
    'quantopian/zipline/zipline/utils/idbox.py',
    'quantopian/zipline/zipline/pipeline/loaders/frame.py',
    'quantopian/zipline/zipline/pipeline/filters/__init__.py',
    'quantopian/zipline/zipline/data/continuous_future_reader.py',
    'quantopian/zipline/zipline/assets/assets.py',
    'quantopian/zipline/tests/test_cmdline.py',
    'quantopian/zipline/zipline/utils/exploding_object.py',
    'quantopian/zipline/zipline/pipeline/loaders/blaze/estimates.py',
    'quantopian/zipline/tests/utils/test_numpy_utils.py',
    'quantopian/zipline/zipline/finance/metrics/tracker.py',
    'quantopian/zipline/zipline/utils/input_validation.py',
    'quantopian/zipline/zipline/utils/paths.py',
    'quantopian/zipline/tests/data/test_minute_bars.py',
    'quantopian/zipline/zipline/utils/data.py',
    'quantopian/zipline/zipline/pipeline/loaders/base.py',
    'quantopian/zipline/zipline/data/bundles/csvdir.py',
    'quantopian/zipline/zipline/data/hdf5_daily_bars.py',
    'quantopian/zipline/zipline/pipeline/mixins.py',
    'quantopian/zipline/tests/pipeline/test_alias.py',
    'quantopian/zipline/zipline/errors.py',
    'quantopian/zipline/zipline/data/bundles/quandl.py',
    'quantopian/zipline/tests/data/test_daily_bars.py',
    'quantopian/zipline/tests/pipeline/test_adjustment.py',
    'quantopian/zipline/tests/utils/daily_bar_writer.py',
    'quantopian/zipline/zipline/assets/asset_db_schema.py',
    'quantopian/zipline/tests/resources/fetcher_inputs/fetcher_test_data.py',
    'quantopian/zipline/zipline/pipeline/dtypes.py',
    'quantopian/zipline/tests/data/test_dispatch_bar_reader.py',
    'quantopian/zipline/zipline/pipeline/graph.py',
    'quantopian/zipline/zipline/pipeline/term.py',
    'quantopian/zipline/tests/test_tradesimulation.py',
    'quantopian/zipline/zipline/country.py',
    'quantopian/zipline/zipline/finance/trading.py',
    'quantopian/zipline/tests/finance/test_risk.py',
    'quantopian/zipline/zipline/pipeline/factors/basic.py',
    'quantopian/zipline/zipline/pipeline/sentinels.py',
    'quantopian/zipline/tests/data/bundles/test_quandl.py',
    'quantopian/zipline/tests/data/test_adjustments.py',
    'quantopian/zipline/tests/history/generate_csvs.py',
    'quantopian/zipline/zipline/pipeline/pipeline.py',
    'quantopian/zipline/zipline/utils/deprecate.py',
    'quantopian/zipline/zipline/data/bundles/core.py',
    'quantopian/zipline/zipline/utils/context_tricks.py',
    'quantopian/zipline/zipline/pipeline/expression.py',
    'quantopian/zipline/zipline/extensions.py',
    'quantopian/zipline/zipline/pipeline/loaders/events.py',
    'quantopian/zipline/zipline/dispatch.py',
    'quantopian/zipline/tests/test_examples.py',
    'quantopian/zipline/zipline/zipline_warnings.py',
    'quantopian/zipline/zipline/finance/ledger.py',
    'quantopian/zipline/tests/pipeline/test_dataset.py',
    'quantopian/zipline/zipline/utils/date_utils.py',
    'quantopian/zipline/tests/pipeline/test_downsampling.py',
    'quantopian/zipline/zipline/finance/__init__.py',
    'quantopian/zipline/zipline/pipeline/factors/technical.py',
    'quantopian/zipline/zipline/utils/preprocess.py',
    'quantopian/zipline/tests/test_data_portal.py',
    'quantopian/zipline/zipline/utils/security_list.py',
    'quantopian/zipline/zipline/pipeline/loaders/earnings_estimates.py',
    'quantopian/zipline/zipline/pipeline/api_utils.py',
    'quantopian/zipline/zipline/utils/classproperty.py',
    'quantopian/zipline/zipline/data/__init__.py',
    'quantopian/zipline/zipline/gens/utils.py',
    'quantopian/zipline/zipline/_version.py',
    'quantopian/zipline/zipline/testing/pipeline_terms.py',
    'quantopian/zipline/tests/data/test_fx.py',
    'quantopian/zipline/zipline/finance/position.py',
    'quantopian/zipline/zipline/algorithm.py',
    'quantopian/zipline/zipline/utils/api_support.py',
    'quantopian/zipline/zipline/pipeline/engine.py',
    'quantopian/zipline/zipline/pipeline/loaders/blaze/events.py',
    'quantopian/zipline/zipline/utils/compat.py',
    'quantopian/zipline/tests/finance/test_cancel_policy.py',
    'quantopian/zipline/zipline/utils/numpy_utils.py',
    'quantopian/zipline/zipline/test_algorithms.py',
    'quantopian/zipline/tests/pipeline/test_computable_term.py',
    'quantopian/zipline/zipline/data/minute_bars.py',
    'quantopian/zipline/tests/utils/test_final.py',
    'quantopian/zipline/zipline/pipeline/factors/__init__.py',
    'quantopian/zipline/zipline/testing/slippage.py',
    'quantopian/zipline/zipline/data/benchmarks.py',
    'quantopian/zipline/tests/pipeline/test_events.py',
    'quantopian/zipline/tests/events/test_events_nyse.py',
    'quantopian/zipline/tests/test_finance.py',
    'quantopian/zipline/tests/test_security_list.py',
    'quantopian/zipline/tests/test_continuous_futures.py',
    'quantopian/zipline/zipline/utils/math_utils.py',
    'quantopian/zipline/tests/resources/pipeline_inputs/generate.py',
    'quantopian/zipline/tests/metrics/test_core.py',
    'quantopian/zipline/zipline/utils/events.py',
    'quantopian/zipline/tests/pipeline/test_engine.py',
    'quantopian/zipline/tests/pipeline/test_statistical.py',
    'quantopian/zipline/zipline/pipeline/data/testing.py',
    'quantopian/zipline/tests/pipeline/test_numerical_expression.py',
    'quantopian/zipline/zipline/utils/memoize.py',
    'quantopian/zipline/zipline/data/in_memory_daily_bars.py',
    'quantopian/zipline/zipline/testing/fixtures.py',
    'quantopian/zipline/zipline/api.py',
    'quantopian/zipline/zipline/pipeline/filters/smoothing.py',
    'quantopian/zipline/zipline/utils/functional.py',
    'quantopian/zipline/zipline/pipeline/visualize.py',
    'quantopian/zipline/zipline/pipeline/loaders/equity_pricing_loader.py',
    'quantopian/zipline/tests/test_bar_data.py',
    'quantopian/zipline/tests/pipeline/test_column.py',
    'quantopian/zipline/tests/utils/test_pandas_utils.py',
    'quantopian/zipline/zipline/assets/futures.py',
    'quantopian/zipline/tests/test_api_shim.py',
    'quantopian/zipline/zipline/finance/blotter/__init__.py',
    'quantopian/zipline/zipline/utils/final.py',
    'quantopian/zipline/tests/pipeline/test_slice.py',
    'quantopian/zipline/zipline/data/adjustments.py',
    'quantopian/zipline/tests/utils/test_sharedoc.py',
    'quantopian/zipline/zipline/pipeline/__init__.py',
    'quantopian/zipline/zipline/pipeline/hooks/delegate.py',
    'quantopian/zipline/zipline/pipeline/loaders/blaze/core.py',
    'quantopian/zipline/zipline/pipeline/loaders/testing.py',
    'quantopian/zipline/tests/utils/test_math_utils.py',
    'quantopian/zipline/zipline/utils/pandas_utils.py',
    'quantopian/zipline/zipline/assets/asset_writer.py',
    'quantopian/zipline/zipline/utils/factory.py',
    'quantopian/zipline/zipline/data/fx/__init__.py',
    'quantopian/zipline/docs/source/conf.py',
    'quantopian/zipline/tests/pipeline/test_international_markets.py',
    'quantopian/zipline/tests/pipeline/test_adjusted_array.py',
    'quantopian/zipline/zipline/__init__.py',
    'quantopian/zipline/zipline/pipeline/loaders/synthetic.py',
    'quantopian/zipline/tests/finance/test_slippage.py',
    'quantopian/zipline/zipline/utils/sqlite_utils.py',
    'quantopian/zipline/zipline/pipeline/factors/statistical.py',
    'quantopian/zipline/zipline/pipeline/loaders/blaze/utils.py',
    'quantopian/zipline/zipline/pipeline/hooks/no.py',
    'quantopian/zipline/zipline/lib/labelarray.py',
    'quantopian/zipline/zipline/examples/buy_and_hold.py',
    'quantopian/zipline/zipline/pipeline/common.py',
    'quantopian/zipline/zipline/pipeline/loaders/blaze/__init__.py',
    'quantopian/zipline/zipline/pipeline/hooks/iface.py',
    'quantopian/zipline/tests/utils/test_argcheck.py',
    'quantopian/zipline/tests/pipeline/test_dtypes.py',
    'quantopian/zipline/tests/pipeline/test_pipeline.py',
    'quantopian/zipline/tests/test_testing.py',
    'quantopian/zipline/zipline/finance/transaction.py',
    'quantopian/zipline/zipline/examples/dual_ema_talib.py',
    'quantopian/zipline/zipline/examples/buyapple.py',
    'quantopian/zipline/zipline/testing/core.py',
    'quantopian/zipline/tests/test_restrictions.py',
    'quantopian/zipline/tests/pipeline/test_technical.py',
    'quantopian/zipline/zipline/pipeline/hooks/__init__.py',
    'quantopian/zipline/zipline/pipeline/downsample_helpers.py',
    'quantopian/zipline/tests/pipeline/test_hooks.py',
    'quantopian/zipline/tests/utils/test_cache.py',
    'quantopian/zipline/zipline/pipeline/domain.py',
    'quantopian/zipline/zipline/testing/debug.py',
    'quantopian/zipline/zipline/utils/run_algo.py',
    'quantopian/zipline/tests/utils/test_preprocess.py',
    'quantopian/zipline/zipline/pipeline/data/equity_pricing.py',
    'quantopian/zipline/zipline/utils/sentinel.py',
    'quantopian/zipline/zipline/finance/controls.py',
    'quantopian/zipline/zipline/finance/constants.py',
    'quantopian/zipline/zipline/finance/shared.py',
    'quantopian/zipline/zipline/sources/benchmark_source.py',
    'quantopian/zipline/zipline/pipeline/loaders/__init__.py',
    'quantopian/zipline/zipline/data/resample.py',
    'quantopian/zipline/zipline/finance/execution.py',
    'quantopian/zipline/zipline/examples/momentum_pipeline.py',
    'quantopian/zipline/tests/pipeline/test_frameload.py',
    'quantopian/zipline/zipline/pipeline/hooks/testing.py',
    'quantopian/zipline/zipline/utils/tradingcalendar.py',
    'quantopian/zipline/zipline/data/loader.py',
    'quantopian/zipline/zipline/pipeline/hooks/progress.py',
    'quantopian/zipline/zipline/finance/metrics/__init__.py',
    'quantopian/zipline/tests/test_algorithm.py',
    'quantopian/zipline/zipline/finance/cancel_policy.py',
    'quantopian/zipline/zipline/utils/dummy.py',
    'quantopian/zipline/tests/pipeline/test_filter.py',
    'quantopian/zipline/zipline/assets/__init__.py',
    'quantopian/zipline/zipline/pipeline/loaders/utils.py',
    'quantopian/zipline/tests/utils/test_metautils.py',
    'quantopian/zipline/zipline/pipeline/factors/factor.py',
    'quantopian/zipline/zipline/utils/pool.py',
    'quantopian/zipline/zipline/assets/exchange_info.py',
    'quantopian/zipline/zipline/gens/tradesimulation.py',
    'quantopian/zipline/tests/pipeline/test_us_equity_pricing_loader.py',
    'quantopian/zipline/zipline/examples/__init__.py',
    'quantopian/zipline/zipline/assets/synthetic.py',
    'quantopian/zipline/zipline/pipeline/classifiers/__init__.py',
    'quantopian/zipline/tests/finance/test_transaction.py',
    'quantopian/zipline/tests/test_benchmark.py',
    'quantopian/zipline/tests/events/test_events.py',
    'quantopian/zipline/tests/pipeline/test_multidimensional_dataset.py',
    'quantopian/zipline/etc/gen_type_stubs.py',
    'quantopian/zipline/tests/utils/test_date_utils.py',
    'quantopian/zipline/zipline/data/fx/base.py',
    'quantopian/zipline/zipline/finance/commission.py',
    'quantopian/zipline/zipline/data/dispatch_bar_reader.py',
    'quantopian/zipline/docs/deploy.py',
    'quantopian/zipline/tests/pipeline/test_quarters_estimates.py',
    'quantopian/zipline/zipline/data/bar_reader.py',
    'quantopian/zipline/tests/utils/test_sentinel.py',
    'quantopian/zipline/zipline/data/fx/hdf5.py',
    'quantopian/zipline/tests/data/bundles/test_csvdir.py',
    'quantopian/zipline/zipline/finance/metrics/metric.py',
    'quantopian/zipline/zipline/utils/calendars.py',
    'quantopian/zipline/zipline/pipeline/data/__init__.py',
    'quantopian/zipline/zipline/data/history_loader.py',
    'quantopian/zipline/zipline/assets/roll_finder.py',
    'quantopian/zipline/zipline/utils/cache.py',
    'quantopian/zipline/etc/conda_build_matrix.py',
    'quantopian/zipline/zipline/utils/enum.py',
    'quantopian/zipline/tests/pipeline/test_factor.py',
    'quantopian/zipline/tests/test_ordering.py',
    'quantopian/zipline/tests/pipeline/base.py',
    'quantopian/zipline/zipline/finance/metrics/core.py',
    'quantopian/zipline/conda/1_setuptools_scm/meta.yaml',
    'quantopian/zipline/conda/requests/meta.yaml',
    'quantopian/zipline/conda/intervaltree/meta.yaml',
    'quantopian/zipline/conda/trading-calendars/meta.yaml',
    'quantopian/zipline/conda/logbook/meta.yaml',
    'quantopian/zipline/conda/0_sortedcontainers/meta.yaml',
    'quantopian/zipline/.appveyor.yml',
    'quantopian/zipline/conda/ta-lib/meta.yaml',
    'quantopian/zipline/.pre-commit-config.yaml',
    'quantopian/zipline/conda/zipline/meta.yaml',
    'quantopian/zipline/conda/lru-dict/meta.yaml',
    'quantopian/zipline/.travis.yml',
    'quantopian/zipline/conda/iso3166/meta.yaml',
    'quantopian/zipline/conda/empyrical/meta.yaml',
    'quantopian/zipline/conda/bcolz/meta.yaml',
    'quantopian/zipline/conda/numexpr/meta.yaml',
    'quantopian/zipline/conda/iso4217/meta.yaml',
    'quantopian/zipline/conda/alembic/meta.yaml',
    'quantopian/zipline/conda/parameterized/meta.yaml',
    'quantopian/zipline/conda/python-interface/meta.yaml',
    'quantopian/zipline/conda/logbook/build.sh',
    'quantopian/zipline/ci/travis/overwrite_requirements.sh',
    'quantopian/zipline/vagrant_init.sh',
    'quantopian/zipline/conda/ta-lib/build.sh',
    'quantopian/zipline/ci/travis/install_miniconda.sh',
    'quantopian/zipline/etc/rebuild-cython.sh',
    'quantopian/zipline/etc/docker_cmd.sh',
    'quantopian/zipline/conda/intervaltree/build.sh',
    'quantopian/zipline/conda/README.md',
    'quantopian/zipline/docs/notebooks/tutorial.ipynb',
    'quantopian/zipline/zipline/examples/buyapple.ipynb'
],
authorName: 'Richard Frank',
authorEmail: 'richafrank@users.noreply.github.com',
commitDate: '2020-10-14T16:36:49Z',
authorAvatarUrl: 'https://avatars.githubusercontent.com/u/2123660?v=4',
authorFollowersUrl: 'https://api.github.com/users/richafrank/followers',
otherLanguages: [
    'YAML',
    'Shell',
    'Markdown',
    'Jupyter'
],
_class: 'com.apollo.merger.model.github.GenericGithubCommits'

hcipek commented 2 years ago

Örnek CommitFile data:

_id: ObjectId('622efcfb22d80003aa111711'),
language: 'Python',
fileName: 'aleju/imgaug/checks/check_withchannels.py',
content: 'from __future__ import print_function, division\n\nimport cv2\nimport numpy as np\nfrom skimage import data\n\nimport imgaug as ia\nfrom imgaug import augmenters as iaa\n\nTIME_PER_STEP = 10000\n\n\ndef main():\n    image = data.astronaut()\n    print("image shape:", image.shape)\n    print("Press ENTER or wait %d ms to proceed to the next image." % (TIME_PER_STEP,))\n\n    children_all = [\n        ("hflip", iaa.Fliplr(1)),\n        ("add", iaa.Add(50)),\n        ("dropout", iaa.Dropout(0.2)),\n        ("affine", iaa.Affine(rotate=35))\n    ]\n\n    channels_all = [\n        None,\n        0,\n        [],\n        [0],\n        [0, 1],\n        [1, 2],\n        [0, 1, 2]\n    ]\n\n    cv2.namedWindow("aug", cv2.WINDOW_NORMAL)\n    cv2.imshow("aug", image[..., ::-1])\n    cv2.waitKey(TIME_PER_STEP)\n\n    for children_title, children in children_all:\n        for channels in channels_all:\n            aug = iaa.WithChannels(channels=channels, children=children)\n            img_aug = aug.augment_image(image)\n            print("dtype", img_aug.dtype, "averages", np.average(img_aug, axis=tuple(range(0, img_aug.ndim-1))))\n\n            title = "children=%s | channels=%s" % (children_title, channels)\n            img_aug = ia.draw_text(img_aug, x=5, y=5, text=title)\n\n            cv2.imshow("aug", img_aug[..., ::-1])  # here with rgb2bgr\n            cv2.waitKey(TIME_PER_STEP)\n\n',
orderLine: '0,5,10,15,20,25,30,35,40,45',
valid: true,
_class: 'com.apollo.merger.model.github.GithubCommitFile'

hcipek commented 2 years ago

http://3.71.207.240:9661 http://3.71.207.240:9671 http://3.71.207.240:9000/ --portainer un:admin, pw:apolloadmin http://3.71.207.240:9001/ --kafdrop http://3.71.207.240:8081/ --mongoExpress un:admin, pw:apollo

ITU-Apollo / data-engineering

Mongo'da ne yaptık akış olarak anlatalım #10