lenmus / lomse

A C++ library for rendering, editing and playing back music scores.
MIT License
117 stars 28 forks source link

How to get the clicked measure according to the xy coordinates of the mouse click. #353

Closed ayyb1988 closed 2 years ago

ayyb1988 commented 2 years ago

I am going to implement the function of touch into the scorescreen to switch to a defined position in the score and play along from there, and and I found the issue#233. Thank you very much. But there are two questions to ask

  1. How to get the clicked measure according to the xy coordinates of the mouse click.
  2. How to highlight the current measure when clicked
cecilios commented 2 years ago

Very interesting question!

Using the measure number to point to a certain place in the score is tricky. Although for most common scores, just providing the measure number does the job, it is necessary to take into account that for polymetric music (music in which not all instruments have the same time signature), the measure number is not an absolute value, common to all score instruments (score parts), but it is relative to an instrument.

Due to this, I didn't I coded in Lomse methods using the measure number. But I'm observing that Lomse users ask for measure number as parameter so I recently started to code methods using measures. Currently there are very few methods using measures and most Lomse methods receiving as parameter a measure number also receives the instrument number to which the measure refers, with a default value of 0 (first instrument). The first measure (anacruxis or not) is always measure=0.

And most methods have an alternative signature using instead a MeasureLocator, an struct for describing any location on the score:

struct MeasureLocator
{
    int iInstr;             //instrument number (0..n)
    int iMeasure;           //measure number (0..m), for the instrument
    TimeUnits location;     //TimeUnits from start of measure

    MeasureLocator() : iInstr(0), iMeasure(0), location(0.0) {}
    MeasureLocator(int i, int m, TimeUnits l) : iInstr(i), iMeasure(m), location(l) {}

};

Now, returning to your questions:

How to get the clicked measure according to the xy coordinates of the mouse click

Currently, there is no a simple way of getting the measure number. A possible way of doing it would be:

//get mouse click point (pixels)
int xm = mouse_x;
int ym = mouse_y;

//get clicked object (e.g. staffObj, auxObj, staff, whatever)
ImoObj* pImo = spInteractor->find_object_at(Pixels(xm), Pixels(ym));

//get score page
int iPage = pGView->page_at_screen_point(double(xm), double(ym));

//get information for positioning a caret at that point
LUnits x(xm);
LUnits y(ym);
spInteractor->screen_point_to_page_point(&x, &y);
DocCursorState state = pGView->click_event_to_cursor_state(iPage, LUnits(x), LUnits(y), pImo, pGmo);

//get measure number and instrument from caret state
SpElementCursorState elmState = state.get_delegate_state();
ScoreCursorState* pState = static_cast<ScoreCursorState*>( elmState.get() );
int iMeasure = pState->measure();
int iInstr = pState->instrument();

Certainly, this is not an acceptable API and it forces to access and to understand some internal objects. I'm going to fix this by including specific API methods, (suggestions welcome!) probably:

//get mouse click point
Pixels x = Pixels( mouse_x );
Pixels y = Pixels( mouse_y );

//get clicked instrument and measure
int iMeasure = spInteractor->find_measure_at(x, y);
int iInstr = spInteractor->find_instrument_at(x, y);
MeasureLocator ml = spInteractor->find_locator_for(x, y);

I'm currently busy with other tasks but hope to have this done along next week.

How to highlight the clicked measure

Currently there are no methods for highlighting one or more measures. The only available option is to place a fragment mark (See https://lenmus.github.io/lomse/classFragmentMark.html). You will find there an example for placing a mark at the desired measure.

My first impression is that adding methods for highlighting one or more measures should be simple. So I'm going to add this to the list of things to do in the coming days.

How to play from a given measure

ScorePlayer object has method ScorePlayer::play_from_measure(iMeasure, ...). See https://lenmus.github.io/lomse/classScorePlayer.html

But parameter iMeasure does not start in 0, but goes from 1 to n. And the method does not receive the instrument number, so it is not appropriate for polymetric scores. I have to fix this.

ayyb1988 commented 2 years ago

Thanks a lot for such a quick reply. I invoke the find_object_at method according to your commit, but the information in data.ml (MeasureLocator) is all 0. That is to say, both pImo->is_scoreobj() and pImo->is_instrument() are false.

In addition, I thought of a method, when parsing xml to imo, record the corresponding measure value, and then pass it when the corresponding gmo is generated, and the corresponding measure can be obtained according to the gmo

cecilios commented 2 years ago

The method in previous commit to my personal repo is not yet ready. Please wait for a PR to lenmus/lomse repo that will include also tests and documentation. I'm really busy this week and probably this will not be ready until the weekend.

In addition, I thought of a method, when parsing xml to imo, record the corresponding measure value, and then pass it when the corresponding gmo is generated, and the corresponding measure can be obtained according to the gmo

Thanks for the suggestion. Many music programs are focused on measures and this works for 99% of music. But these programs have problems to deal with music without measures (e.g many Eric Satie piano pieces) or polymetric music. Lomse treats measures in a different way and the measure number is not data associated neither to each Imo object nor to each Gmo object.

ayyb1988 commented 2 years ago

thank you very much

cecilios commented 2 years ago

PR #355 , already merged, adds method to find measure number. See details and example in the description of this PR.

Now I will implement a method to highlight one or more measures.

ayyb1988 commented 2 years ago

I ran several test cases from lomse_test_interacotr.cpp For example TEST_FIXTURE(InteractorTestFixture, click_info_604) But the obtained MeasureLocator.iMeasure =-1 , MeasureLocator.ilnst=-1, MeasureLocator.location=0

cecilios commented 2 years ago

These test are not strong. They require that the test runner is build with variable LOMSE_FONTS_PATH pointing to Bravura music font. I had to fix CMakeLists.txt to fix this issue with the tests. Could you please do two things?:

  1. Rebuild Lomse library and the test runner. If after doing this the tests still fail, then
  2. Send me the result of the the test runner, including the initial lines with the paths used.

On a real test, when clicking on an score, have you observed any bad behavior (bad ClickPointData info)?

ayyb1988 commented 2 years ago

First

Rebuild Lomse library and the test runner. If after doing this the tests still fail, then

Modify and run the unit tests as follows

  bool check_click_data2(ClickPointData& data, int iInstr, int iStaff, int iMeasure,
        TimeUnits location, const string& pImoName, const char* name)
        {
            bool fTestOk = data.ml.iInstr == iInstr
                           && data.iStaff == iStaff
                           && data.ml.iMeasure == iMeasure
                           && is_equal_time(data.ml.location, location)
                           && (pImoName=="nullptr" ? data.pImo==nullptr
                                                   : data.pImo->get_name() == pImoName);

//            if (!fTestOk) //      -->Comment out this judgment in order to output the log
            {
                cout << "Error in test " << name << endl
                     << "    Result: "
                     << "instr=" << data.ml.iInstr << ", staff=" << data.iStaff
                     << ", measure=" << data.ml.iMeasure << ", location=" << data.ml.location
                     << ", pImo is " << (data.pImo==nullptr ? "nullptr"
                                                            : data.pImo->get_name()) << endl
                     << "  Expected: "
                     << "instr=" << iInstr << ", staff=" << iStaff
                     << ", measure=" << iMeasure << ", location=" << location
                     << ", pImo is " << pImoName <<endl;
            }
            return fTestOk;
        }

    TEST_FIXTURE(InteractorTestFixture, click_info_604)
    {
        //@604. Click point on instrument brace

        LomseDoorway doorway;
        doorway.init_library(k_pix_format_rgb24, 82);
        LibraryScope libraryScope(cout, &doorway);
        libraryScope.set_default_fonts_path(TESTLIB_FONTS_PATH);
        Presenter* pPresenter = doorway.open_document(k_view_vertical_book,
            m_scores_path + "07012-two-instruments-four-staves.lmd");
        Interactor* pIntor = pPresenter->get_interactor_raw_ptr(0);

        //get graphic model clicked object
        LUnits x = 1634.16;
        LUnits y = 7208.22;
        GraphicModel* pGM = pIntor->get_graphic_model(); //this also forces to engrave the score
        GmoObj* pGmo = pGM->hit_test(0, x, y);

        //get clicked point info
        ClickPointData data = GModelAlgorithms::find_info_for_point(x, y, pGmo);

        CHECK( check_click_data2(data, -1, -1, -1, 0, "instrument", test_name()) );  //--> invoke check_click_data2
    }

The result of running is as follows:

Running all tests

Error in test click_info_604 Result: instr=-1, staff=-1, measure=-1, location=0, pImo is instrument Expected: instr=-1, staff=-1, measure=-1, location=0, pImo is instrument

Success: 2134 tests passed.

Second

On a real test, when clicking on an score, have you observed any bad behavior (bad ClickPointData info)?

I tested it with wxwidget(viewTypes-wx.cpp) and found that what I got was also bad ClickPointData info( instr=-1, staff=-1, measure=-1, location=0). details as follows:

void MyCanvas::on_mouse_event(wxMouseEvent& event)
{
    if (!m_pPresenter) return;

    if (SpInteractor spInteractor = m_pPresenter->get_interactor(0).lock())
    {
        wxEventType nEventType = event.GetEventType();
        wxPoint pos = event.GetPosition();
        unsigned flags = get_mouse_flags(event);

        if (nEventType==wxEVT_LEFT_DOWN)
        {
            flags |= k_mouse_left;
            spInteractor->on_mouse_button_down(pos.x, pos.y, flags);

            // --> test find_info_for_point     start-----
            Interactor* pIntor = m_pPresenter->get_interactor_raw_ptr(0);

            LUnits x = pos.x;
            LUnits y = pos.y;
            GraphicModel* pGM = pIntor->get_graphic_model(); //this also forces to engrave the score
            GmoObj* pGmo = pGM->hit_test(0, x, y);

            //get clicked point info
            ClickPointData data = GModelAlgorithms::find_info_for_point(x, y, pGmo);
            ImoObj *pObj = data.pImo;
            MeasureLocator ml = data.ml;
            if (ml.is_valid())
            {
                //clicked point is a valid location on an score
                int iMeasure = ml.iMeasure;
                int iInstr = ml.iInstr;
                int iStaff = data.iStaff;

                //do whatever you like with this information

                cout << "iMeasure:"<<iMeasure<<" iInstr:"<<iInstr<< "iStaff:"<<iStaff <<endl;
            }
        }
      // ---> test find_info_for_point    end-----
        ...
   }
cecilios commented 2 years ago

I do not have enough information to understand this problem.

Unit tests

Please after building Lomse just run unit tests without modifying the tests code. Just do:

cd path-to-build-folder-containig-the-binaries
./testlib

You shoud get something as:

Lomse version 0.29.0-50+9618bab2. Library tests runner.

Lomse build date: 11-May-2022 17:07:07
Path for tests scores: '/datos/cecilio/lm/projects/lomse/trunk/test-scores/'
Tests fonts path: '/datos/cecilio/lm/projects/lomse/trunk/fonts/'
Lomse fonts path: '/usr/local/share/lomse/fonts/'

Running all tests

Success: 2134 tests passed.
Test time: 12.52 seconds.

Process returned 0 (0x0)   execution time : 12.534 s
Press ENTER to continue.

Please send me the output you get.

Test with an app

You don't need to replicate the test code. In a real application just do:

ClickPointData data = spInteractor->find_click_info_at(x, y);

Please, replace these lines in you test code:

// --> test find_info_for_point     start-----
        Interactor* pIntor = m_pPresenter->get_interactor_raw_ptr(0);

        LUnits x = pos.x;
        LUnits y = pos.y;
        GraphicModel* pGM = pIntor->get_graphic_model(); //this also forces to engrave the score
        GmoObj* pGmo = pGM->hit_test(0, x, y);

        //get clicked point info
        ClickPointData data = GModelAlgorithms::find_info_for_point(x, y, pGmo);
        ImoObj *pObj = data.pImo;
        MeasureLocator ml = data.ml;
        if (ml.is_valid())
        {
            //clicked point is a valid location on an score
            int iMeasure = ml.iMeasure;
            int iInstr = ml.iInstr;
            int iStaff = data.iStaff;

            //do whatever you like with this information

By this code:

// --> test find_info_for_point     start-----
        ClickPointData data = spInteractor->find_click_info_at(x, y);
        ImoObj *pObj = data.pImo;
        MeasureLocator ml = data.ml;

        //------- To dump some values ------------------
            //clicked point info
        cout << "x:" << x << ", y:" << y << ", iMeasure:" << iMeasure << ", iInstr:" << iInstr << ", iStaff:"
             << iStaff << ", pImo is " << (data.pImo==nullptr ? "nullptr" : data.pImo->get_name()) << endl;
        double xPos = double(x);
        double yPos = double(y);
        int iPage = page_at_screen_point(xPos, yPos);
        screen_point_to_page_point(&xPos, &yPos);
        cout << "LUnits: x=" << xPos << ", y=" << yPos << ", iPage=" << iPage << endl;

            //dump of graphic model
        GraphicModel* pGM = spInteractor->get_graphic_model();
        int pages = pGM->get_num_pages();
        cout << "Dump of graphic model:" << endl;
        for (int i=0; i < pages; ++i)
        {
            cout << "Page " << i
                 << " ==================================================================="
                 << endl;
            pGM->dump_page(i, cout);
        }
        cout << endl << endl;
        //------- End of dump -------------------------------------

        if (ml.is_valid())
        {
            //clicked point is a valid location on an score
            int iMeasure = ml.iMeasure;
            int iInstr = ml.iInstr;
            int iStaff = data.iStaff;

            //do whatever you like with this information

And send me the output. Thank you.

cecilios commented 2 years ago

Analyzing more in detail test click_info_604 I observe that the output you sent to me it no longer fails. After rebuilding the library it seems that all test passed ok. Could you please confirm this?

As to the real app test, I'm assuming that after you rebuild Lomse, you also re-installed it and then you re-build the application (viewTypes-wx.cpp). Is this correct?

ayyb1988 commented 2 years ago
Analyzing more in detail test click_info_604 I observe that the output you sent to me it no longer fails. 
After rebuilding the library it seems that all test passed ok. Could you please confirm this?

Yes, the lomse unit tests ran successfully

As to the real app test, I'm assuming that after you rebuild Lomse,
 you also re-installed it and then you re-build the application (viewTypes-wx.cpp). Is this correct?

According to your example above, Use spInteractor->find_click_info_at(x, y); get valid ClickPointData correctly. thank you very much

cecilios commented 2 years ago

You are welcome! Can I close this issue?

ayyb1988 commented 2 years ago
highlight one or more measures.

Any suggestions or references for this?

cecilios commented 2 years ago

Ah! True! I can not close this issue yet. I've started to work on this, but I'm having a really busy month. Probably the highlight method could be ready along next week.

cecilios commented 2 years ago

Closed in PR #356. Usage example:

ClickPointData data = pInteractor->find_click_info_at(pos.x, pos.y);
Document* pDoc = pPresenter->get_document_raw_ptr();
ImoScore* pScore = dynamic_cast<ImoScore*>( pDoc->get_content_item(0) );

if (data.ml.is_valid() && pScore)
{
    ImoId scoreId = pScore->get_id();

    MeasureHighlight* mark = pInteractor->add_measure_highlight(scoreId, data.ml);

    .. now refresh the window so that the added marker is displayed
}

See API documentation: https://lenmus.github.io/lomse/classMeasureHighlight.html https://lenmus.github.io/lomse/classInteractor.html#add8f800d6e000146c0888b5c1160fe7e