search

EMCECS / ecs-sync

ecs-sync is a bulk copy utility that can move data between various systems in parallel
Apache License 2.0
60 stars 22 forks source link

Questions about data flow and sequence of the work #106

Closed imtester1 closed 7 months ago

imtester1 commented 7 months ago

I'm working on a migration using ECS-SYNC and I have a question.

I am curious about the order of operations and data flow.

First, the order of work is

  1. Create a Clip-ID list.
  2. Create an xml file.
  3. Start the task with the ecs-sync-ctl --submit command. I understand it.

The data flow is

  1. The sync server requests Centera with the clip-id list.
  2. Centera sends data to the sync server.
  3. The sync server delivers the received data to ecs.
  4. Verify the data in the above process.
  5. Done

Am I understanding this correctly?

hoelgu commented 7 months ago

Hi

Lot’s of things are going to be happening in parallel and not in a sequence.

Once the first clips are in the database, ECSSync begins to read them from the source, write them to the target, read them from the target, compare the MD5 values to ensure that nothing has changed.

Make sure to check the enhanced db entries checkbox so that you have the MD5 values and can prove that nothing has changed.

If you still migrate off of Centera, make sure to watch the number of connections on the access nodes, typically we have seen roughly 2.5 to 3 times as many connections on the access nodes as threads configured in ECSSync. You can easily overwhelm a small Centera with a single ECSSync VM.

In case of any questions, reach out to @.***

Take care, Holger

Holger Jakob Head of Unstructured Data Solutions

Tel. <tel:+41315113600> +41 79 415 56 22

https://www.backup.ch/

From: jyjung05 @.> Sent: Friday, 15 December 2023 06:28 To: EMCECS/ecs-sync @.> Cc: Subscribed @.***> Subject: [EMCECS/ecs-sync] Questions about data flow and sequence of the work (Issue #106)

I'm working on a migration using ECS-SYNC and I have a question.

I am curious about the order of operations and data flow.

First, the order of work is

  1. Create a Clip-ID list.
  2. Create an xml file.
  3. Start the task with the ecs-sync-ctl --submit command. I understand it.

The data flow is

  1. The sync server requests Centera with the clip-id list.
  2. Centera sends data to the sync server.
  3. The sync server delivers the received data to ecs.
  4. Verify the data in the above process.
  5. Done

Am I understanding this correctly?

— Reply to this email directly, view it on GitHub https://github.com/EMCECS/ecs-sync/issues/106 , or unsubscribe https://github.com/notifications/unsubscribe-auth/AZKNYXAB7OJWFA3FL5FOSUTYJPNWBAVCNFSM6AAAAABAV5NAHSVHI2DSMVQWIX3LMV43ASLTON2WKOZSGA2DEOJSGQ3DMMA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

imtester1 commented 7 months ago

During the migration process, is Centera's data directly transferred to ECS without going through the ECS-Sync server?

hoelgu commented 7 months ago

You could achieve that with a native transformation. I find ECSSync more controllable and less risky to overload the Centera, especially if it is a small one.

Reach out to @.*** if we can be of any assistance

Holger

Holger Jakob +41 79 415 56 22

On 17 Dec 2023, at 13:06, testmeyout @.***> wrote:



During the migration process, is Centera's data directly transferred to ECS without going through the ECS-Sync server?

— Reply to this email directly, view it on GitHubhttps://github.com/EMCECS/ecs-sync/issues/106#issuecomment-1859152425, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AZKNYXDUTCQJSIM2HPKQWOTYJ3N2NAVCNFSM6AAAAABAV5NAHSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJZGE2TENBSGU. You are receiving this because you commented.Message ID: @.***>

imtester1 commented 7 months ago

Thanks for you