search

SitecoreUnicorn / Unicorn

A Sitecore utility designed to simplify deployment of Sitecore items across environments automatically
MIT License
269 stars 116 forks source link

Initial Serialization Clearing Entire DataStore #381

Closed jesslich423 closed 4 years ago

jesslich423 commented 4 years ago

We're on Sitecore 8.2 Update 7 using Unicorn version 3.3.2. Upon creating a new configuration and performing the initial serialization, the entire serialized folder is being cleared.

markgibbons25 commented 4 years ago

Do you mean the serialization folder of that configuration? Or is it clearing all others? Either way, you should upgrade to the latest Unicorn 4 release which supports all Sitecore versions.

jesslich423 commented 4 years ago

It's clearing all the other folders. Thanks, I will definitely start with upgrading, but I'm thinking it might also be a configuration issue.

cassidydotdk commented 4 years ago

For that, I would need to see the configuration. However as mentioned, you really need to upgrade. Unicorn 3.3.2 is almost 4 years old.

jesslich423 commented 4 years ago

Unicorn.config:

false
jesslich423 commented 4 years ago 
<!--
    Unicorn.config

    This file configures the Unicorn serialization system, which enables automatic
    synchronization of Sitecore items without packages or manual merging.

    This file should be present on all environments Unicorn is present on. It is
    safe to leave on Content Delivery servers, as it changes no stock Sitecore
    configuration.

    http://github.com/kamsar/Unicorn
-->
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <unicorn>
            <!--
                Configurations

                These define sets of configuration that Unicorn can run. For example, you might
                want to serialize certain items differently, or segregate configurations so
                that critical ones can run separately from non-essential ones to save time.

                See the README here for more information:
                https://github.com/kamsar/Unicorn/blob/master/README.md

                If you're familiar with DI and IoC containers, each configuration is effectively
                a dependency container which inherits from the global container defined in
                <defaults>.

                NOTE: Your own configurations should be defined in config patch files. See
                Unicorn.Configs.Default.example for a baseline you can make your own from.
            -->
            <configurations>
            </configurations>

            <!--
                These are the default Unicorn dependency configurations. These affect all
                configurations, unless specifically overridden in the configuration definition.

                Each dependency type can have non-dependency constructor params passed to it by
                adding XML attributes to the main declaration - e.g. <foo type="..." bar="hello">
                would pass "hello" to public MyType(string bar)

                You can also receive any XML body passed to the dependency to a 'configNode'
                XmlElement parameter. This is how the SerializationPresetPredicate defines its
                preset.

                Did you know you can define your own global dependencies and override them in
                configurations? You can!
            -->
            <defaults>
                <!-- The source data store is where we read data from. Usually, this would be Sitecore. -->
                <sourceDataStore type="Rainbow.Storage.Sc.SitecoreDataStore, Rainbow.Storage.Sc" singleInstance="true"/>

                <!--
                        The target data store is where we write serialized items to.

                        Note the target data store's rootPath can be any of:
                        Absolute filesystem path, e.g. c:\foo\bar
                        Web-root-relative path, e.g. ~/data/serialization or ~/../out-of-root-serialization
                        Data-folder-relative path, e.g. $(dataFolder)\serializedItems

                        DO NOT SHARE A ROOT PATH BETWEEN CONFIGURATIONS (at least if you're using SFS).
                        They can clobber each other's folders. You may inject the name of the current
                        configuration as a variable with $(configurationName).

                        The data cache uses a memory cache to store serialized items read from disk. It
                        is recommended if using transparent syncing for performance. It's not really
                        needed otherwise.
                    -->
                <targetDataStore physicalRootPath="~/../../serialization/$(configurationName)" useDataCache="false" type="Rainbow.Storage.SerializationFileSystemDataStore, Rainbow" singleInstance="true"/>

                <serializationFormatter type="Rainbow.Storage.Yaml.YamlSerializationFormatter, Rainbow.Storage.Yaml" singleInstance="true">
                    <fieldFormatter type="Rainbow.Formatting.FieldFormatters.MultilistFormatter, Rainbow" />
                    <fieldFormatter type="Rainbow.Formatting.FieldFormatters.XmlFieldFormatter, Rainbow" />
                    <fieldFormatter type="Rainbow.Formatting.FieldFormatters.CheckboxFieldFormatter, Rainbow" />
                </serializationFormatter>

                <deserializer type="Unicorn.Deserialization.UnicornDeserializer, Unicorn" ignoreBranchId="true" singleInstance="true" />
                    <deserializerLogger type="Unicorn.Deserialization.DefaultDeserializerLogger, Unicorn" singleInstance="true"/>

                <evaluator type="Unicorn.Evaluators.SerializedAsMasterEvaluator, Unicorn" singleInstance="true"/>
                    <evaluatorLogger type="Unicorn.Evaluators.DefaultSerializedAsMasterEvaluatorLogger, Unicorn" singleInstance="true"/>

                <!-- The ItemComparer handles comparing items for the Evaluator and SerializationConflictProcessor -->
                <itemComparer type="Rainbow.Diff.ItemComparer, Rainbow" singleInstance="true">
                    <!--
                      You may add your own field comparisons here to determine equality - they are
                      evaluated in order and the first one to say it can compare gets the job.

                      Note: the DefaultComparison, which does a string Equals() comparison, is
                      automatically registered last and does not need to appear here.
                    -->
                    <fieldComparer type="Rainbow.Diff.Fields.XmlComparison, Rainbow" />
                    <fieldComparer type="Rainbow.Diff.Fields.MultiLineTextComparison, Rainbow" />
                    <fieldComparer type="Rainbow.Diff.Fields.MultilistComparison, Rainbow" />
                    <fieldComparer type="Rainbow.Diff.Fields.CheckboxComparison, Rainbow" />
                </itemComparer>

                <!-- The default predicate is set here so that configurations need not specify its class name to use it. -->
                <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true" />

                <!--
                    The field filter can be used to ignore fields when comparing or serializing (i.e. don't write them to disk).
                    Commonly, metadata fields such as Last Updated will be ignored to prevent SCM conflicts.
                -->
                <fieldFilter type="Rainbow.Filtering.ConfigurationFieldFilter, Rainbow" singleInstance="true">
                    <exclude fieldID="{B1E16562-F3F9-4DDD-84CA-6E099950ECC0}" note="'Last run' field on Schedule template (used to register tasks)" />
                    <exclude fieldID="{52807595-0F8F-4B20-8D2A-CB71D28C6103}" note="'__Owner' field on Standard Template" />
                    <exclude fieldID="{8CDC337E-A112-42FB-BBB4-4143751E123F}" note="'__Revision' field on Standard Template" />
                    <exclude fieldID="{D9CF14B1-FA16-4BA6-9288-E8A174D4D522}" note="'__Updated' field on Standard Template" />
                    <exclude fieldID="{BADD9CF9-53E0-4D0C-BCC0-2D784C282F6A}" note="'__Updated by' field on Standard Template" />
                    <exclude fieldID="{001DD393-96C5-490B-924A-B0F25CD9EFD8}" note="'__Lock' field on Standard Template" />
                </fieldFilter>

                <!-- 
                    Controls configuration-level sync settings

                    UpdateLinkDatabase: If true, the links will be updated for synced items after the sync has completed. If false (default) links are not updated for performance.
                        Updating links is important if you are syncing user-facing content where link tracking is important. It is not very important for system items (templates/renderings).

                    UpdateSearchIndex: If true, the search index(es) containing the item will be updated with item changes after the sync has completed. If false, indexing will not be updated for performance.
                        Updating the index is important for content that relies on indexing, which may include most user-facing content items. Most of the time templates and renderings don't need indexing.

                    NOTE: UpdateLinkDatabase and UpdateSearchIndex also apply to items that are reloaded from disk when using Transparent Sync, as well as normal Sync.
                -->
                <syncConfiguration updateLinkDatabase="false" updateSearchIndex="false" type="Unicorn.Loader.DefaultSyncConfiguration, Unicorn" singleInstance="true" />

                <!-- Note that the DebugSerializationLoaderLogger is also available if you want more detailed logging and timing information. -->
                <loaderLogger type="Unicorn.Loader.DefaultSerializationLoaderLogger, Unicorn" singleInstance="true"/>

                <loaderConsistencyChecker type="Unicorn.Loader.DuplicateIdConsistencyChecker, Unicorn"/>
                    <loaderConsistencyCheckerLogger type="Unicorn.Loader.DefaultDuplicateIdConsistencyCheckerLogger, Unicorn" singleInstance="false"/>

                <loaderDeserializeFailureRetryer type="Unicorn.Loader.DeserializeFailureRetryer, Unicorn" singleInstance="false"/>

                <dataProviderConfiguration enableTransparentSync="false" type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" singleInstance="true" />
                    <dataProviderLogger type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderLogger, Unicorn" singleInstance="true"/>

                <logger type="Unicorn.Logging.PubSubLogger, Unicorn" singleInstance="true" />

                <syncCompleteDataCollector type="Unicorn.Pipelines.UnicornSyncComplete.DefaultSyncCompleteDataCollector, Unicorn" singleInstance="true" />
            </defaults>

            <!--
                The configuration provider wires up how the configuration(s) Unicorn is running are parsed.
                The default provider reads values from this config section.
            -->
            <configurationProvider type="Unicorn.Configuration.ConfigyConfigurationProvider, Unicorn"/>
        </unicorn>

        <settings>
            <!--
                Unicorn will log warnings when Transparent Sync is enabled and Fast Query is used (which cannot resolve any Transparent Sync items).
                If you are getting log spammed by this, and really want to turn it off, you can enable this setting. You naughty person. Quit using fast query! :)
            -->
            <setting name="Unicorn.DisableFastQueryWarning" value="false" />
        </settings>
        <pipelines>
            <!-- Run before a sync operation of one or more configurations. Handlers should implement IUnicornSyncStartProcessor -->
            <unicornSyncStart>
            </unicornSyncStart>

            <!-- Run before an individual configuration starts sync. Handlers should implement IUnicornSyncBeginProcessor -->
            <unicornSyncBegin>
                <processor type="Unicorn.Pipelines.UnicornSyncBegin.ResetSyncCompleteDataCollector, Unicorn" />
            </unicornSyncBegin>

            <!-- Run after an individual configuration completes sync. Handlers should implement IUnicornSyncCompleteProcessor -->
            <unicornSyncComplete>
                <processor type="Unicorn.Pipelines.UnicornSyncComplete.DictionaryCacheClearer, Unicorn" />
                <processor type="Unicorn.Pipelines.UnicornSyncComplete.SyncedItemPostProcessor, Unicorn">
                    <!--
                        Set this to "true" if you are experiencing odd behaviour when syncing a large number of configurations. 
                        It will hurt performance significantly however.
                    -->
                    <UnconditionalCacheClearing>false</UnconditionalCacheClearing>
                </processor>
                <processor type="Unicorn.Pipelines.UnicornSyncComplete.SyncCompleteLogger, Unicorn" />
            </unicornSyncComplete>

            <!-- Run after all configuration(s) complete sync in a batch. Handlers should implement IUnicornSyncEndProcessor -->
            <unicornSyncEnd>
                <processor type="Unicorn.Pipelines.UnicornSyncEnd.SendSerializationCompleteEvent, Unicorn" />
            </unicornSyncEnd>

            <!-- Run before a reserialize operation of one or more configurations. Handlers should implement IUnicornReserializeStartProcessor -->
            <unicornReserializeStart>
            </unicornReserializeStart>

            <!-- Run after a configuration completes reserialization. Handlers should implement IUnicornReserializeCompleteProcessor -->
            <unicornReserializeComplete>
            </unicornReserializeComplete>

            <!-- Expands variables in configuration XML - e.g. Helix conventions -->
            <unicornExpandConfigurationVariables>
                <!-- Expands variables of the format $(configurationName). Useful for setting defaults by convention e.g. in abstract configurations -->
                <processor type="Unicorn.Pipelines.UnicornExpandConfigurationVariables.ConfigurationNameVariablesReplacer, Unicorn" />
                <!-- Expands variables of the format $(layer) or $(module) based on configuration naming conventions. Useful for setting defaults by convention e.g. in abstract configurations -->
                <processor type="Unicorn.Pipelines.UnicornExpandConfigurationVariables.HelixConventionVariablesReplacer, Unicorn" />
            </unicornExpandConfigurationVariables>
        </pipelines>

        <processors>
            <saveUI>
                <!-- Prevent fake 'overwrite?' warnings when using transparent sync due to how it handles revisions -->
                <processor mode="on" type="Unicorn.Pipelines.SaveUi.TransparentSyncAwareCheckRevision, Unicorn" patch:instead="processor[contains(@type, 'CheckRevision')]"/>
            </saveUI>
        </processors>

        <sitecore.experienceeditor.speak.requests>
            <!-- Prevent fake 'overwrite?' warnings when using transparent sync due to how it handles revisions -->
            <request patch:instead="*[@name='ExperienceEditor.Save.CheckRevision']" name="ExperienceEditor.Save.CheckRevision" type="Unicorn.ExperienceEditor.Speak.Ribbon.Requests.SaveItem.TransparentSyncAwareCheckRevision, Unicorn"/>
        </sitecore.experienceeditor.speak.requests>

        <sites>
            <!-- A controlled site context to run Unicorn sync processes in, where we know Language Fallback features and other misguidings are inactive -->
            <site name="unicorn"
                inherits="shell"
                enableItemLanguageFallback="false"
                enableFieldLanguageFallback="false"
                enforceVersionPresence="false" />
        </sites>
    </sitecore>
</configuration>
jesslich423 commented 4 years ago

Unicorn.Default:

jesslich423 commented 4 years ago 
<!--
    Unicorn.Configs.Default.config

    This is an example of how to configure a basic Unicorn configuration using your own config patch file.
    Copy this file to use as a basis for your own configuration definitions.

    Enabled configuration definition patches should be present on all environments Unicorn is present on.

    See Unicorn.config for commentary on how configurations operate, or https://github.com/kamsar/Unicorn/blob/master/README.md
-->
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <unicorn>
            <configurations>
                <!--
                    The default configuration is an example to start making other configurations from.

                    WHAT SHOULD I INCLUDE?
                    In general, include the fewest items possible. This both makes things faster and reduces the amount of things kept in source control.
                    The most common candidates for serialization are layout items and template items, but Unicorn can serialize any type of item or field including media.

                    BEHAVIORS
                    Configurations can override the default dependencies defined in Unicorn.config's <defaults> to apply
                    custom behaviors to specific configurations. They behave like an inherited IoC container.

                    DEPENDENCIES
                    Configurations may also depend on each other. Add a comma-delimited list of configuration names to depend on to the 'dependencies' attribute on the configuration.
                    Dependent configurations do not force dependencies to sync if not selected, but syncs will always occur in dependency order if multiple dependent configurations sync at once.
                    Transitive dependency and multiple dependency (comma delimited) are supported.

                    TIPS
                    Favor using several configurations over a single monolithic one.
                    Favor using more includes and fewer excludes in your predicates.
                    Start with minimal configurations and add includes as you need to serialize new things.
                -->

        <configuration name="Layouts" description="Layouts" dependencies="">
          <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
            <include name="Layouts" database="master" path="/sitecore/layout/Layouts/Custom" />
          </predicate>
          <dataProviderConfiguration enableTransparentSync="true" type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" singleInstance="true" />
        </configuration>
        <configuration name="Templates" description="Templates" dependencies="Layouts">
          <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
            <include name="Templates" database="master" path="/sitecore/templates/Custom" />
          </predicate>
          <dataProviderConfiguration enableTransparentSync="true" type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" singleInstance="true" />
        </configuration>
        <configuration name="Renderings" description="Renderings" dependencies="Layouts">
          <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
            <include name="Renderings" database="master" path="/sitecore/layout/Renderings/Custom" />
          </predicate>
          <dataProviderConfiguration enableTransparentSync="true" type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" singleInstance="true" />
        </configuration>
        <configuration name="Workflows" description="Workflows" dependencies="Layouts">
          <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
            <include name="Workflows" database="master" path="/sitecore/system/Workflows/Custom" />
          </predicate>
          <dataProviderConfiguration enableTransparentSync="true" type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" singleInstance="true" />
        </configuration>
        <configuration name="Web" description="Configurations for Web" dependencies="Templates">
          <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
            <!--
                            PREDICATE

                            The predicate controls what items are included in the configuration.

                            Each include can also exclude specific subitems in various ways. For a reference of the most current predicate grammar, consult the tests here:
                            https://github.com/kamsar/Unicorn/blob/master/src/Unicorn.Tests/Predicates/TestConfiguration.xml

                            NOTE: after changing what is included or excluded, you should reserialize all items, or at least the added items for additions.
                            NOTE: the "name" attribute controls the folder name the items will go into. If unspecified, the last path segment is used. Names must be unique across the configuration.
                            NOTE: You cannot use excludes with Transparent Sync. See https://github.com/kamsar/Unicorn/wiki/The-Transparent-Sync-Guide
                        -->

            <include name="PlaceholderSettings" database="master" path="/sitecore/layout/Placeholder Settings/Custom" />
            <include name="MediaLibraryDev" database="master" path="/sitecore/media library/Web_Dev" />
            <include name="BranchTemplates" database="master" path="/sitecore/templates/Branches/Custom" />
            <include name="BranchEmailTemplates" database="master" path="/sitecore/templates/Branches/Email Campaign/Messages/Custom Email" />
            <include name="StandardGlobalContent" database="master" path="/sitecore/content/StandardGlobal" />
            <include name="PubTargets1" database="master" path="/sitecore/system/Publishing targets/Pub1" />
            <include name="PubTargets2" database="master" path="/sitecore/system/Publishing targets/Pub2" />
            <include name="WFFMCustomFieldsDev" database="master" path="/sitecore/system/Modules/Web Forms for Marketers/Settings/Field Types/Custom" />
            <include name="WFFMCustomActionsMarketingOptIn" database="master" path="/sitecore/system/Modules/Web Forms for Marketers/Settings/Actions/Save Actions/MarketingOptInUpdate" />
            <include name="WFFMCustomActionsFlushContact" database="master" path="/sitecore/system/Modules/Web Forms for Marketers/Settings/Actions/Save Actions/Flush Contact to xDB" />
            <include name="WFFMFormsRoot" database="core" path="/sitecore/content/Documents and settings/All users/Start menu/Programs/Web Forms for Marketers/Create a New Form" />
            <include name="Sitecore.Support.130279" database="core" path="/sitecore/client/Speak/Layouts/Renderings/Common/PageCode" />
            <include name="ItemBucketsSettings" database="master" path="/sitecore/system/Settings/Buckets/Item Buckets Settings" />
            <include name="SitecoreLanguages" database="master" path="/sitecore/system/Languages" />
            <include name="BlogBucketingRule" database="master" path="/sitecore/system/Settings/Rules/Definitions/Elements/Bucketing/ResolvePublishDateBasedPath" />
            <include name="PersonalizationRules" database="master" path="/sitecore/system/Settings/Rules/Definitions/Elements/Custom Rules" />
            <include name="TaskCommandSync" database="master" path="/sitecore/system/Tasks/Commands/Custom"/>

            <!--<include database="master" path="/sitecore/system/Languages" />
                        <include database="master" path="/sitecore/system/Tasks">
                            <exclude path="Schedules" />
                        </include>
                        <include database="master" path="/sitecore/system/Workflows" />
                        <include database="master" path="/sitecore/system/Settings" />-->
                    </predicate>

                    <!--
                        SYNC TYPE

                        Traditional Sync (the default) updates the state of the database only when a sync operation is run. It supports additional operations but can be more of a chore to remember to sync.
                        Transparent Sync (preferred) updates the state of Sitecore instantly as soon as changes to files occur. It is optimal for development purposes, but has a few limitations.
                        See the guide to help decide: https://github.com/kamsar/Unicorn/wiki/The-Transparent-Sync-Guide
                    -->
                    <dataProviderConfiguration enableTransparentSync="true" type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" singleInstance="true" />

                    <!-- 
                        SYNC CONFIGURATION

                        UpdateLinkDatabase: If true, the links will be updated for synced items after the sync has completed. If false (default) links are not updated for performance.
                            Updating links is important if you are syncing user-facing content where link tracking is important. It is not very important for system items (templates/renderings).

                        UpdateSearchIndex: If true, the search index(es) containing the item will be updated with item changes after the sync has completed. If false, indexing will not be updated for performance.
                            Updating the index is important for content that relies on indexing, which may include most user-facing content items. Most of the time templates and renderings don't need indexing.

                        MaxConcurrency: Controls how many threads Unicorn can use when syncing this configuration. Concurrency can usually increase sync speed 30-50% over single threading.
                            IF YOUR CONFIGURATION INCLUDES TEMPLATES YOU MUST SET THIS TO 1 OR YOUR SYNC WILL HANG INDEFINITELY DUE TO A SITECORE ISSUE
                            IF YOU ARE ON SITECORE 8.0 U2 OR EARLIER, YOU MUST SET THIS TO 1 DUE TO A SITECORE ISSUE

                            This value can be set to significantly higher than the number of CPU cores, as these are largely I/O bound.
                            Use fewer threads for HDDs or slow SQL servers, and more threads for SSDs.
                            For a SSD workload on a quad-core CPU 16 seems to be a decent number.

                        NOTE: UpdateLinkDatabase and UpdateSearchIndex also apply to items that are reloaded from disk when using Transparent Sync, as well as normal Sync.
                    -->
                    <syncConfiguration updateLinkDatabase="true" updateSearchIndex="true" maxConcurrency="1" type="Unicorn.Loader.DefaultSyncConfiguration, Unicorn" singleInstance="true" />
                </configuration>

            </configurations>
        </unicorn>
    </sitecore>
</configuration>
jesslich423 commented 4 years ago

Test.config

jesslich423 commented 4 years ago 
<!--
    Unicorn.Configs.Default.config

    This is an example of how to configure a basic Unicorn configuration using your own config patch file.
    Copy this file to use as a basis for your own configuration definitions.

    Enabled configuration definition patches should be present on all environments Unicorn is present on.

    See Unicorn.config for commentary on how configurations operate, or https://github.com/kamsar/Unicorn/blob/master/README.md
-->
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <unicorn>
            <configurations>
                <configuration name="Test" description="Test Initial Sync" dependencies="">
                    <predicate type="Unicorn.Predicates.SerializationPresetPredicate, Unicorn" singleInstance="true">
                        <include name="Test" database="master" path="/sitecore/content/Global" />
                    </predicate>
                    <dataProviderConfiguration enableTransparentSync="true" type="Unicorn.Data.DataProvider.DefaultUnicornDataProviderConfiguration, Unicorn" singleInstance="true" />
                </configuration>
            </configurations>
        </unicorn>
    </sitecore>
</configuration>
jesslich423 commented 4 years ago

seems like it's most likely due to the targetDataStore setting, but looks ok to me..

jesslich423 commented 4 years ago

it looks like the $(configurationName) variable isn't properly being replaced

jesslich423 commented 4 years ago

I figured it out.. there was a patch config for the physicalRootPath that I missed that doesn't contain the $(configurationName) param. Thanks!